As a veteran Adobe Analytics Implementation Consultant, I have experienced over and over various pain points when tasked with maintaining an implementation. Sometimes the pain points are due to insufficient resource allocation and sometimes it’s due to simply not knowing how to properly care for an implementation. In this post, I will be addressing the most common pain points of maintaining an analytics implementation.
Things keep breaking
Symptoms
Reports show more and more “unspecified”. Data analysts surface new tracking issues regularly. You feel like you’re playing whack-a-mole with tracking bugs. The team is losing confidence in the data.
Causes
Lack of data monitoring and maintenance resources allocated.
Treatment
Adobe Analytics requires continuous care so that your data is reliable and actionable at a moment’s notice.
- Ongoing monitoring – set up alerts and regularly review reports for anomalies. Recognize areas with frequent issues. For example, Single Page Applications (SPAs) may be particularly troublesome for your properties.
- Log and document bugs – keep a detailed log of data issues including start and end date, descriptions, and links to Analysis Workspace reports. Additionally, create a “Data Events” shared calendar that is up-to date with events that impact the reports.
- Update passive QA monitoring tools to keep up with product releases
- Design and follow a rigorous QA process for every product release – even releases without analytics bugs. For example, perform a QA regression on staging environment and check the staging report suite before signing off on a release.
Analytics bugs are accumulating
Symptoms
The ticket backlog keeps growing with more and more data tickets. Additionally, Analytics data integrity is gradually deteriorating at an inverse rate to the number of data discrepancy issues arising.
Causes
Insufficient engineering and analytics resources.
Treatment
In every analytics bug ticket be sure to note down Business Impact. The business impact should be the top-most section of the ticket where you explain how the bug has affected the business and how the impact will grow if it continues unaddressed. For example, “The business is unable to measure and optimize main menu links which is critical at this time since the main menu redesign is planned for the next quarter.” Be sure to partner with your business intelligence analyst for impact.
Prioritization will play a key role in any ticket backlog. Be sure to carefully consider the business impact before setting priority on your analytics bug ticket. If unsure of the priority, ask the person or team who will be impacted. Lean on your stakeholders to also vocalize their priorities and impact so that priority is well understood by the product team.
Other tactics to help your tickets get pulled in:
- Categorize or label tickets for easy grouping
- Periodically review the data ticket backlog to check for relevance as the product evolves
Data Issues are discovered months after they start
Symptoms
Has it ever happened to you that you load a report and a number looks off? Then you trend the data and find that tracking broke a month ago? This is the primary way that most people find out something is broken.
Causes
Lack of data monitoring.
Treatment
The only treatment available is to monitor your data and you can achieve this either actively or passively.
Active Monitoring
- Set a routine to regularly review an analytics health report. For example, set up a Monday morning coffee meeting for yourself to open up an Analysis Workspace report and review the data that came in for the previous week. Look at week-over-week changes by each and every dimension and metric. Careful not to leave any out because these will be the ones to surprise you in a month.
- Perform data regressions on every product release candidate. Validate tracking data beacons for your entire implementation. Oftentimes, analytics tracking can break even if a product release doesn’t include any analytics tracking tickets.
- For every release candidate, review staging data in a dev report suite as some bugs will only surface on the reporting end. For example, processing rules may introduce tracking bugs that will not be visible via beacon validation.
Passive Monitoring
- Set up Adobe Alerts. In Analysis Workspace, right-click on a data point and select “Create alert from selection”. Then you can either enter your work email or set up a slack email address for a designated Adobe Analytics Alerts channel.
- Use CanaryIQ. CanaryIQ is a dedicated anomaly detection platform built for Adobe Analytics and uses Machine Learning to learn what is and what isn’t a data anomaly.
- Use a Data Feeds solution. Methods vary greatly here as the data pipeline can differ from company to company. Consult with your Data Engineering team if you have Data Feeds set up already.
- Subscribe to a beacon validation service like Observepoint. Observepoint can be set up to crawl through your site and check analytics tags for all pages. Alternatively, you can set up Observepoint Journeys to check each and every beacon parameter by page/event.
That wraps up the most common Adobe Analytics implementation pain points. Comment below on what you think or if you have other common pain points you would like to share.