Drea Thomas is the engineering director responsible for the front end technology at SnapLogic – which explains his clear bias toward Designer and all things visual.
When you build a data integration platform, it is easy to expect integrations to look like something from a demo:
Figure 1: DemoLand
But, as we’ve learned over the years, real-world integrations often are a lot messier and can look more like:
- Figure 2: The Real World
The Spring ‘12 release of the SnapLogic Server is all about improving the user experience dealing with large and complex pipelines. The goal is to reduce the time required to build a complex pipeline by an order of magnitude. Let me outline some of the improvements we’ve made to the product to address this.
Latency
In previous versions of the product, calculating pass-through fields (something that is quite complex) was done on the server side. This makes a lot of sense because you can centralize the code in one place and re-use it for Designer, or for any application that builds pipelines using SNAPI. But, when your pipelines look like RealWorld, the pipeline definition is quite large – sometimes several megabytes. Sending it back and forth over the wire introduces significant latency.
In Spring ‘12, we’ve moved some of the most frequently used server logic into the client to greatly reduce latency. In some cases, this leads to a 300% improvement in Designer performance.
Debugging
In DemoLand, every pipeline runs the first time. In the RealWorld, things don’t always work out that way. In previous versions of the product, when a pipeline resource was badly configured, or there was an error in a query or an expression, you would sometimes discover it only when you ran your pipeline. This required you to read an error log to identify which resource caused the problem and why.
In DemoLand identifying the offending resource and fixing it is fairly straightforward because you only have a handful of resources. In RealWorld, even finding the offending resource on the pipeline canvas is challenging.
In Spring ‘12 we’ve done a number of things to improve the situation.
Improved Validation
Components now catch many more problems at validation time – before you even run the pipeline. And the problems that are caught produce better error messages. This tells the developer which resource failed and why.
Resource Highlighting
When a resource fails during pipeline execution, we mark it red and call out the specific error in a tooltip.
Getting Unstuck
In DemoLand, all pipelines run quickly and produce the results you expect. In the RealWorld, pipelines can take far longer than anticipated or produce unexpected results. When you want to identify the bottleneck in a pipeline, you need to see how much time is being spent by which resource and what it is doing.
Previous versions of the product allowed the user to view aggregate statistics at the end of a pipeline run, which gave a good indicator. But what do you do when a pipeline hangs, or takes a really long time to complete? In Spring ‘12, we now show resource statistics in real time during pipeline execution in Designer.
Data Tracing
Previous versions of the product allowed you to generate trace files showing you the value of records and they flowed through the pipeline. This enabled you to figure out why your join was not producing the result you expected. But, the trace data was written to files and required you to view the data outside of the Designer.
In Spring ‘12, you can now generate and view trace data directly within Designer.
- Pipeline Trace Data
This allows you to identify problems without having to switch tools, making debugging faster and easier.
In addition to the GUI improvements, there are many, many other improvements and bug fixes. See the release notes for a comprehensive list.
Whether we’ve achieved our “order of magnitude” goal is up to our customers to judge. But, as someone who builds a lot of pipelines, I can say that Spring ‘12 has made my life much easier.
I’m very proud of this release, and I’d encourage everyone to give it a try and send us feedback.
-Drea-
Continuing our series of guest posts by partners, this week we are happy to have Tara Gilliam, a lead developer at Anaplan discuss her experience in building the Anaplan Snap:
Thanks, Praneal!
We’ve been working with SnapLogic to provide Anaplan integration for a mutual customer who is a heavy user of FinancialForce, Xactly, Salesforce, Zuora and several other cloud data services. With SnapLogic’s cloud integration platform, we’ve produced a full-featured Snap, allowing customers to transfer data between Anaplan and all the data services SnapLogic already supports.
The SnapLogic platform makes it quick to start creating value. The Anaplan Snap was my first project as a new developer at Anaplan. I worked initially on uploading data from SnapLogic into Anaplan, and after starting from scratch the first beta of the Anaplan Importer was handed over for testing just 3 weeks later.
We’ve worked closely with SnapLogic to add more complexfeatures to the Anaplan Snap. We’ve had quick responses throughout, and their support has been an essential part of our Snap development success. In addition to data upload, our Snap now supports data download back into SnapLogic for processing and verification, and the ability to run independent Imports, Exports, and Actions within Anaplan. This is all supported through SnapLogic’s simple graphical Designer, allowing Anaplan workspace browsing and task configuration through drop-down selections.
The completed Snap is being delivered to SnapLogic for QA this week, and will be available shortly in their online SnapStore. We’re excited about the infinite integration possibilities now available to our customers.
For me, January always tends to be a month for collecting my thoughts after the holidays. These last few weeks, I’ve been reflecting on all we accomplished at SnapLogic in 2011, and thinking about what I’m most excited for in 2012.
I agree with Gaurav Dhillon, our CEO, that this will be “the year of cloud in the enterprise”. Notice how he uses the words “cloud in the enterprise” vs. “enterprise cloud”. That’s because for large enterprises, the cloud is being adopted in a hybrid manner. Hybrid in terms of WHERE it’s being adopted (both Private and Public) and WHO is driving the adoption (both centralized and shadow IT). The bottom line is that enterprises now face a burgeoning number and diversity of new end points within their IT ecosystem.
With all the groundwork we laid in 2011, there’s never been a more exciting time to be at SnapLogic. Put simply, the more things that need to be connected together, the more benefit SnapLogic provides to the customer and the more value we add to their business. Of course, we can’t do it alone and we are looking for great people in all areas of the company.
(This is the first in a three part series)
Joining our discussion is Christophe Barriolade CEO of Orchestra Networks (a leading Master Data Management Vendor) and I’m Zeb Mahmood, Principal Product Marketing Manager MDM at SnapLogic
Master data is the DNA of every company, defining key business objects such as products, reference tables, parties, assets, employees and financial hierarchies across several complex information systems, both on-premise and in the Cloud. Owned and shared by all lines of business, this “single version of the truth” is the building block of every critical business transaction.
Unfortunately, traditional MDM projects are expensive, require years to implement and integrate, and often fail because they are too slow to deliver value. A successful MDM program starts with the business and keeps them in the loop, from the definition of a data model to the governance of master data over time.
On December 8th, 2011 SnapLogic and Orchestra Networks launched the first Cloud MDM solution, offering a web-based, self-service experience to business users, delivering results in just weeks.You can register or view the webinar here to see the solution in action.
The Cloud MDM service, called smartdatagovernance.com, leverages the best of Orchestra Network’s EBX platform and SnapLogic’s Enterprise Cloud Integration Platform.
smartdatagovernance.com is a collaborative master data authoring and governance service that allows you to manage the full life cycle of your master data. You can:
- Define data models and validation controls (business rules)
- Manage any type of data and hierarchies
- Define roles and access rights
- Configure workflow for changes and approvals
- Manage versions
Most enterprises today are a hybrid mix of SaaS/Cloud and on premise applications. The burgeoning number and diversity of Cloud Apps means you’re going to need to quickly integrate more applications into your corporate MDM. SnapLogic drastically reduces the time and effort required to connect all these endpoints to your MDM.
Simply download the free smartdatagovernance.com Snap into SnapLogic to start moving data in and out of your Cloud MDM. You’ll enjoy:
- Improved data load times, leading to higher availability of MDM for business users
- Higher performance of MDM, as data integration and transformation tasks are offloaded
- Support for a vast array of data formats and protocols. This is significant as each endpoint that MDM connects to may speak a different language in terms of:
- Format: CSV, XML, JSON, etc.
- Transport Protocol: JDBC, SOAP, REST, etc.
- Frequency: Real-time, batch (hourly, nightly, weekly), etc.
- Delta: Attribute/Field level delta, record level delta, full feed, etc.
- Simplified implementation architecture, as SnapLogic can be the gatekeeper for all incoming and outgoing data feeds
Of course there’s much more. Click here to learn how Cloud MDM can empower business users across multiple departments to achieve faster ROI and greater business agility.
(This is the final post in a three-part series)
Over the past couple weeks, I’ve shared my thoughts on the integration challenges of the last 20 years. All of these have led to where our industry is today. Here at SnapLogic, we observed the various impediments to integration, and set off a few years ago to tackle cloud-to-cloud and cloud-to-ground integration in a new way.
We wanted to help virtualize your data outside your cloud providers to give you the greatest amount of abstraction and freedom of choice in how you choose to leverage your IT portfolio. We also felt passionate about bringing integration to the mere mortal. We believed that integration needed to be quick, relativity simple, and most importantly it must have a cost point radically lower than in the past. We asked ourselves, could we do for integration what Salesforce.com did for CRM?
So what happened?
- First, we developed the first and only REST based integration platform. REST makes the web work, and it gives SnapLogic the scale and resilience we have all marveled at with the Web.
- Second, we broke the code on a “Snap” architecture that truly allows us to bring all end points into a neutral canonical format.
- Third, we made this Snap architecture available to third parties such as integrators and private companies to build commercial and private Snaps, much like the Apple App Store model which has revolutionized the mobile consumer Internet. This creates a marketplace of available Snaps which keeps the cost for integration low.
- Finally, we developed a 100% visual environment that allows people to create integrations, resulting in a commercial integration platform that allows customers solve their integration challenges at 25% of the cost and time required with legacy products.
Here’s where it gets really interesting, because a funny thing happened on the way to solving the cloud integration problem. In our effort to make cloud integration easy, we actually just made overall integration easy and cost effective!
Remember the retailer I mentioned in my first post? Their goal is to eliminate hand coding and free up those 50 FTEs for more strategic work, by leveraging SnapLogic as their integration framework for everything between existing initiatives best-suited for ESB or ETL implementations. Current integration approaches are slow and cost prohibitive relative to Cloud apps and enterprise are looking for a lightweight solution to orchestrate business processes across Cloud and ground assets. We see the same pattern among our media customers. Between ETL and EAI/ESBs, there’s a gap in the integration infrastructure of most organizations, and a pressing need for a commercial platform that can sit side by side with ESB and ETL infrastructure.
How about you? Do you find that current integration approaches are slow and cost prohibitive relative to your Cloud apps?