Changelog

This tool is still in active development (see below!) and is by no means the finished article - it's a continually evolving skunk works project. Apologies is things break from time to time! It is intended for only the hardiest of early adopters to get a feel for. Any feedback or suggestions will be gratefully received!

What's new?

Update 13/03/17
GeoIP: Surfaced GeoIP output to show the geographic location of where the website resides.
Back-end: Cleared up some log message notice's and warnings.
Update 12/03/17
GeoIP: Built GeoIP import scripts.
Status: Fixed SQL errors on reporting.
Update 09/03/17
Markup: Added preconnect tags to the document head.
Setup: Added updated set of user agents.
Update 04/03/17
Status: Updated stats for 2017
Update 24/02/17
Dashboard: Can now detect what CMS is being used, from 15 of the most common systems.
Images: Added an SVG logo library to the project, containing flags for every country and region across the globe, which can be used to enhance the UI.
Images: Added an SVG country flag library to the project, containing approx 1000 logos, which can be used to enhance the UI.
Email: Notification emails now include a small thumbnail screenshot of the website being audited.
Dashboard: Added links that are discovered in <link> tags to the broken link checker report.
Update 19/02/17
Dashboard: Added website improvement suggestions to the overview and score pages.
Update 14/02/17
Email: Recommendations added to email
Score: Improved data structure to handle scores
Update 13/02/17
Dashboard: It is now possible to load the next 10 result in any given report, rather than download the whole report.
Score: Added new data structure to support explanations of how the scores are deduced.
Score: Added loads more detail to the score page so that users can see where they need to improve aspects of the site.
Score: Factored broken image src's into the score
Update 06/02/17
Security: Tightened up access to parts of the application.
Preferences: Added extra protection around options such as "Delete My Account".
Preferences: Now shows what membership tier a user is on. This is also visible in the top nav.
Payments: Integrated the application with Stripe to manage payments and subscriptions.
Plan page: Create a pricing / plan page.
Update 03/02/17
Homepage: Improved responsiveness.
Update 28/01/17
Dashboard: Added an area line-chart type for illustrating things such as pages at depth.
Dashboard: Added support for customisable axis label on graphs via the rel attribute.
Update 26/01/17
Homepage: Fixed some responsive issues so the navigation works better on mobile and tablet devices.
Dashboard: Graph to show distribution of image types.
Dashboard: Graph to show pages at a given depth.
Dashboard: Graph to show content types of pages.
Dashboard: Misc graph additions throughout.
Update 25/01/17
Crawl: Now capturing the full HTTP header response for every webpage.
Crawl: Storing full HTTP header response in a dynamic structured format that allows for integration into existing reports.
Dashboard: Made first version of the HTTP header responses visible on the report.
Update 08/01/17
Dashboard: Fixed HTTP analysis links from 'Issues' area into the actual reports.
Update 04/01/17
Crawl: Added better support for discovered compressed and executable files.
Emails: Bug fix for email notifications.
Update 28/12/16
Bug: Fixed issue where a site homepage could sometimes be excluded from an audit. Thanks to Jesse Skeens for reporting.
Bug: Improved favicon detection
Dashboard: Improved UI, so that empty columns are hidden on certain tables.
Update 26/12/16
Bug: Better handling of internal URIs that start with //
Dashboard: New report for images - showing images by type e.g. 100 image/jpeg, 50 image/gif etc.
Dashboard: New report for images - showing favicons by type e.g. 100 image/vnd.microsoft.icon, 100 image/png etc.
Dashboard: New report for images - showing unique favicons
Update 22/12/16
Crawl: Fixed bug with how paths beginning with // were resolving.
Dashboard: New report showing total weight of assets (images, scripts and stylesheets), by page.
Dashboard: Added the image, style and stylesheet size metrics to the performance area.
Dashboard: Removed the 'misc data points' from the Overview section.
Update 21/12/16
Crawl: Now capturing metrics on responses of Script and Stylesheet assets. Also the original SRC and HREF values.
Dashboard: New report on Broken Script SRC's.
Dashboard: New report on Total Weight of Scripts, by page.
Dashboard: New report on Largest Single Scripts.
Dashboard: New report on Broken Stylesheet HREF's.
Dashboard: New report on Total Weight of Stylesheets, by page.
Dashboard: New report on Largest Single Stylesheets.
Update 13/12/16
Email: Much improved automated email notifications and messaging.
Email: Created HTML email template.
Email: Cleared out legacy Mandrill code, in favour of AWS SES.
Responsive CSS: Improvements made throughout.
Update 11/12/16
Crawl: Now capturing the original image SRC attribute, and the resolved path.
Dashboard: Cleaned up handling of reporting of images that use a data: type within an image SRC.
Dashboard: New report to show images that use a data: type within an image SRC
Dashboard: Extended image reporting to show example page urls of pages that have image problems.
Dashboard: Added new report to show the total weight of images, per page URL.
Dashboard: Fixed image report that shows pages that have images without any alt text set.
Email: Improved the emails that the tool auto-generates, by making them HTML responsive.
Update 28/11/16
Dashboard: New report to show linked asset sizes summary, so that the raw HTML page weight summary can be seen separately.
Dashboard: Further enhanced the Image reporting, with example source URLs
Dashboard: New report to show number of pages per language type discovered
Dashboard: Fixed UI for when investigating what pages a spelling mistake is on.
Update 27/11/16
Contact: Fixed the form. Dodgy logic was preventing the contact messages coming through.
Upgraded Jquery to 3.1.1
Dashboard: Next to each URL, there is now a button that allows you to inspect the HTTP Headers.
Dashboard: WHO IS lookup now works for HTTPS protocol. This was a bug fix.
Dashboard: Updated and added various copy on the reporting screen.
Setup: Added additional validation and user feedback for domains which can't be crawled.
Back-end: Cleaned up code to prevent a variety of log messages being recorded for PHP notices and warnings.
Update 21/11/16
Crawl: Now capturing the full HTTP response from image files
Dashboard: Showing full HTTP response for image files
Dashboard: Showing largest images discovered on the site.
Update 17/11/16
Crawl: Fixed bug around restricting audits to a given language. The 'all' option was behaving inconsistently.
Update 16/11/16
Back-end: Improved debugging information.
Update 30/9/16
Theme: Changed the default theme to something less default BootStrap based.
Content: Tweaked copy throughout.
Menu: Added 'Features' Menu item.
Update 20/9/16
Dashboard: Invalid internal URI's are now tested for
Update 12/9/16
Dashboard: Fixed the internal link references for the duplicate content reports i.e. it now correctly lists all pages that are discovered with the same duplicate content
Dashboard: Fixed the Canonicalised domain check, by performing deeper iterative redirect checks
Dashboard: Added report to show all Open Graph tags
Dashboard: Now collecting and showing hreflang tags on the audit reports
Dashboard: Now collecting and showing DNS Prefetch tags on the audit reports
Update 03/9/16
Dashboard: Added metrics for largest and smallest page sizes that have discovered
Update 02/9/16
Thumbnail generation throughout is now automatic if cached versions are not detected
Update 27/8/16
Dashboard: Fixed bug with generating the website screenshots. Screenshots are now also auto-trimmed to get rid of surplus white space.
Dashboard: Generated full page image screenshot of the target page which can be downloaded by users.
Email: Check to ensure users are verified before sending audit report emails.
Update 26/8/16
Dashboard: Added fastest and slowest page performance metrics.
Dashboard: Added shortest and longest page metrics.
Update 25/8/16
Added more usage statistics to the administration usage page.
Update 24/8/16
Upgraded various JQuery plugins, DataTables, and tweaked scripts to support JQuery 3.
Update 16/8/16
Upgraded Bootstrap to 3.3.7
Upgraded JQuery to 3.1
Upgraded Highcharts to 4.2.6
Update 16/7/16
Crawl: Audits can now be restricted to any given language, providing that the html tag has the lang attribute set e.g. <html lang="en">
Back-end: Now capturing information about the pages language which can be used in analysis
Bug: Fixed the contact form
Update 24/6/16
Back-end: Improved capture of scores, so that analysis is more efficient.
Dashboard: New report that captures redirections upon entry to a site.
Dashboard: Improved rendering rules of data points, and forced 404 output.
Score: Factored in the number of redirections upon entry to a site.
Score: Factored in the how the site handles forced 404 errors.
Setup: Added a new minimum debug focused crawl that can be run with a single click.
Crawl: Can now capture variables at any point during the crawl, and report generation process.
Update 8/6/16
Back-end: Improved debugging of crawls where there is a lot of redirection upon initially hitting the domain.
Update 4/5/16
Back-end: Added email test facility
Update 3/5/16
Back-end: Now sending emails through Amazon SES
Install: Process for installing the application for the first time is improved, with more feature detection and better messaging.
Update 25/4/16
Status: Optimised administration reporting queries. Reporting section now loads 75% quicker.
Update 21/3/16
Setup: Can now configure if links discovered in frames, iframes and image map area tags should be crawled.
Score: 'F' grade score colour fixed.
Update 15/3/16
Crawl: Major bug fix on multi-threaded crawls that resulted in partially downloaded pages.
Setup: Added crawl depth feedback so that users can better understand the progress of a crawl.
Update 14/3/16
Back-end: Improved ability to cancel crawls.
Back-end: Improved how, and the frequency in which data is saved to disc.
Back-end: Improved how audit progress tables are saved to disc.
Report: Fixed report that shows the Queued links.
Report: Fixed issue with reporting information of site error handling and favicon presence for target urls entered without a trailing /.
Crawl: Fixed bug that caused the crawler to ignore nofollow instructions on multi-threaded crawls
Crawl: Fixed bug that caused links to sometimes be omitted from a multi-threaded crawl
Update 12/3/16
Back-end: Code clean-up and commenting in reporting library.
Content: Polished the copy throughout the information pages.
Setup: Admin helpers to do a quick full crawl of a site.
Update 11/3/16
Crawl: Fixed detection of mailto: and javascript: where case wasn't lower.
Status: Added source user IP analysis
Status: Added source user join date analysis
Status: Added source user last login date analysis
Update 2/3/16
Setup: Fixed issue where something the report wouldn't show, if you closed the browser windows during the audit.
Update 29/2/16
Back-end: Integrated xhprof into the application for improved performance analysis when developing locally.
Dashboard: Fixed caching issue for when you force the regeneration of a thumbnail
Status: Added more application statistics
Status: Improved the users contacts UI
Update 24/2/16
My reports: Improved UI.
Update 23/2/16
Dashboard: Improved UI rendering of tooltips.
Dashboard: Improved UI rendering of linking pages (when you click the icon).
Front-end: Upgraded to jQuery 2.2.1
Dashboard: Fixing 'copy' functionality and added fallback support from browsers that do not have Flash installed / enabled
Update 20/2/16
Admin: Improved application install process.
Update 19/2/16
Crawl: Fixed bug relating to capture of email address using the mailto: protocol in a href. Instances of this where it contained a ? resulted in an error.
Dashboard: Fixed bug relating to direct links into # tab anchors
Dashboard: Upgraded HighCharts to 4.2.3
Homepage: Added carousel block with more information around what the tool can do
Update 18/2/16
Dashboard: The "My Reports" and "Showcase Reports" dropdowns are now alphabetical, and unrestricted in length.
Crawl: Audits can now be run with the shortest urls analysed first
Crawl: Audits can now be run with the longest urls analysed first
Front-end: Upgraded to jQuery 2.2.0
Front-end: Upgraded to DataTables to 1.10.11
Back-end: Upgraded to QueryPath to 3.0.4
Update 12/2/16
Upgraded to PhantomJS 2.1
Dashboard: Fixed issue with certain reports missing download links
Dashboard: Fixed notice in CSV downloads alerting of 'undefined variable'
Update 10/2/16
Fixed issue with latest version of CURL that prevented gzipped HTML being read properly.
Update 3/1/16
File system: Fixed all file naming conventions that aren't Windows compatible.
Update 2/1/16
Crawl: Prevented the crawling of certain types of url extensions.
Crawl: Prevented the crawling of certain domains.
Update 21/12/15
Crawl: Improved crawl failure messaging.
Status: Banning by activity, based on IP is active.
Status: Can add IP's to a ban list through the admin UI for either crawls, messages or both.
Status: Implemented the DB structure for the IP banning functionality to counter spam crawls and messaging.
Update 4/12/15
Status: Optimised report table loading.
Update 30/11/15
Back-end: Added new dictionary database table containing all English swear words.
My Reports: Fixed bugs relating to incorrect parameter being used on the dashboard links.
Crawl: Can now crawl pages that return a 303 HTTP code.
Dashboard: New Content Report - Scans website text for any occurrences of "lorem ipsum" copy.
Update 26/11/15
Back-end: Cleanup of library file structure.
Upgraded Bootstrap to 3.3.6
Update 05/11/15
Back-end: Can now manipulate global application settings via the admin UI.
Reports: Improved mark-up so that it makes better use of Bootstrap defaults
Reports: Fixed bug on spelling pane when bringing in full content
Reports: Added a case toggle on the spelling pane
Update 04/11/15
Back-end: Added global settings table to allow for better application configuration management.
Back-end: Began migrating hard-coded settings to settings table to allow for manipulation via admin UI.
Update 08/10/15
Back-end: DataTables can now load in data from the server.
Back-end: Migrated all usages of DataTables to load data from the server, as opposed to pre-loading it on page load.
Update 07/10/15
Back-end: Set up new SQL queries to allow DataTables to load additional data via AJAX.
Update 25/09/15
Dashboard: Update HighCharts library to 4.1.8.
Update 23/09/15
Status: Integration of Google Analytics reports into the administration view.
Update 22/09/15
Status: Better feedback for administration tasks i.e. number of audits affected.
Update 14/09/15
Forgot Password: Improved validation.
Update 08/09/15
My Reports: Now with option to delete old reports.
Dashboard: Removed needless visualisation value in misc data points.
Dashboard: Changed misc data points to be a table.
Dashboard: Links from the overview section, now go to the right "pane" and right report.
Update 01/09/15
Dashboard: Updated copy.
Information pages: Updated copy with some of the newer reports and features, such as spell checking.
Content: Turned the tool on ourselves and fixed a load of spelling mistakes!
Updated DataTables to 1.10.9
Server: Overhauled the .htaccess file to contain latest updates and best practices.
Update 29/08/15
Dashboard spelling fix: prevent concatenation of words
Update 26/08/15
Crawl: Custom dictionary for spell checking
Dashboard: Surfaced first basic report for spell checking
Dashboard: Functionality to show which pages spelling mistakes occur on
Dashboard: Words that potentially contain spelling mistakes can be looked-up on an online dictionary
Update 25/08/15
Dashboard: Fixed bug where same domain links, on a different protocol were being treated as external links.
Score: Integrated no HTTPS in to the score
Dashboard: Fixed bug with generation of the full link data download of a site
Dashboard: New report that lists internal absolute URLs
Dashboard: New report that lists internal relative URLs
Dashboard: Removed the Social and SEO reports because they aren't really relevant to the tool
Dashboard: Added support for Google Tag Manager in the Analytics detection
Crawl: Added pspell library in preparation for spell checking functionality
Crawl: Back-end for spell checking, and suggestion functionality
Crawl: DB structure to handle spell checking
Update 24/08/15
Dashboard: Fixed domain canonicalisation bug
Dashboard: Improved UI with a close button on the fixed LHN
Dashboard: Tests if a site has a HTTPS version or not
Update 06/07/15
Upgraded Bootstrap to 3.3.5
Dashboard: Fixed queued links report.
Audit email notification: Added some high-level audit result information.
Update 12/06/15
Dashboard: Added report for broken image SRC's
Dashboard: Added report for HTTP Status codes of external sites and domains
Score: Factored non-200 http status code external links
Update 19/05/15
Improved admin status screen with minor UI tweaks
Email alerts for 'admin' users when new contact messages are received.
Main Navigation tweaks
Update 10/05/15
Updated jQuery to 1.11.3
Update 25/04/15
Back-end: Code cleanup after running static analysis tool over code base.
Update 21/04/15
Users: Created an upgraded standard user account role type that can crawl up to 5000 urls.
Setup: Updated user agent settings.
Setup: Added option to use a proxy service for pro users.
Back-end: Cleanup of obsolete jQuery and Bootstrap library files.
Updated DataTables to 1.10.6
Update 29/03/15
Setup: New option to force ASCII crawl mode.
Setup: Detect if a site is behind a http auth: 401 error code, and alert user. Also handles other http status codes that prevent an audit from happening.
Update 27/03/15
Bug fix: Set the initial referrer to https://metaforensics.io/ as some sites were rejecting the request with a HTTP 400 code.
Update 21/03/15
Upgraded Bootstrap to 3.3.4
Update 17/03/15
Dashboard: New visualisation - External links, by domain, with and without social networks.
Update 13/03/15
Dashboard: Fixed bug with visualisations.
Score: Factored in multiple occurrences / types of web analytics code into the score calculation.
Update 12/03/15
Content: Updated the Recommended SEO Tools page.
Update 01/03/15
Setup: UI tweaks, along with the introduction of a "Tour" feature.
Update 26/02/15
Crawl: Increased the speed of the crawling process by a factor of 10!
Update 20/02/15
Dashboard: New downloadable sitemap.xml files.
Update 19/02/15
Dashboard: New downloadable CSV report for all internally linked pages.
Dashboard: New report for Page Rank of internal pages.
Update 18/02/15
Dashboard: Added "copy" feature to "Pages Linking To" feature
Update 17/02/15
Dashboard: Fixed bug relating to "Pages Linking To" feature
Update 30/01/15
Dashboard: New reports for documents and media assets
Update 29/01/15
Fix crawl bug that wasn't handling URL queries properly
Update 26/01/15
Dashboard: New Developer Helper - Download URLs of each HTTP type as an Array
Update 19/01/15
Upgraded Bootstrap to 3.3.2
Update 04/01/15
Dashboard: New report for viewing canonical <link>
Dashboard: New report for viewing shortlink <link>
Dashboard: New report that shows the number of pages at each depth level within the site
Update 03/01/15
Audit settings: Anonymous users now get a 25 page audit, and registered users get a 500 page audit. Copy updated to reflect this.
Dashboard: HTML markup improvements
Update 02/01/15
Email: Added email notifications to platform that let users know once an audit it complete.
Email: Improved email content.
Email: Added DKIM and SPF records + verified the domain. Auto-generated emails now come from info@metaforensics.io.
Setup: Improved the UX to let users know what the application is actually doing, rather than just asking them to be patient. This is important for larger audits.
Platform: Separated the Report generation for the report viewing.
Update 01/01/15
Dashboard: 404 Error Response code detection report for directories, html files, and scripts.
Update 31/12/14
Platform: Full site wide HTTPS now integrated
Dashboard UI: Addition of single click button, to scroll to top of page.
Update 30/12/14
Dashboard: Added RSS feed detection and reporting to the 'Content' section
Update 29/12/14
Crawl: Identified a major problem with how relative URL's are resolved, and fixed it. How embarrassing.
Homepage: Better resizing of the page when using different themes
Update 27/12/14
Crawl: Can now extract links from <area>, <iframe> and <frame> tags
Update 26/12/14
UI: Improved UI of the preferences page
UI: Added functionality that allows users to choose their own UI theme on the preferences page
Update 23/12/14
Updated jQuery to 1.11.2
Update 10/12/14
Updated DataTables to 1.10.4
Dashboard: Zoomable screenshot thumbnails, along with optimised resolutions for them.
Update 9/12/14
Dashboard: New report that shows internal links with invalid URL's, specifically those with more than one question mark.
Dashboard: Added tooltip helpers to the various icons used throughout
Update 7/12/14
Crawl: Can now handle stylesheets in this format: <style>@import url("stylesheet.css");</style>
Update 6/12/14
Dashboard: Fixed bug on the website thumbnail generation whereby homepage redirects were not handled properly.
Dashboard: Upgraded phantomjs to 1.9.8
Dashboard: Added a radar chart for the score visualisation
Crawl: Fixed bug relating to the crawl not following 301's properly
UI: Content markup tweaks to take advantage of some of the newer Bootstrap features
UI: Added scores to the report examples
Update 29/11/14
Crawl: Fixed bug that caused CDN asset URI's that contain '//' at the start to be reported as an internal link.
Update 21/11/14
Upgraded Bootstrap to 3.3.1
Update 25/10/14
Homepage: Content tweaks and responsive enhancements
Update 24/10/14
Crawl Options: Whether or not cookies should be accepted on a crawl can now be toggled.
Update 21/10/14
Users can now cancel an audit from the setup screen.
Update 15/10/14
Set the production site to use a CDN for Bootstrap and jQuery.
Update 09/10/14
Content UI: Added breadcrumbs to content pages.
Update 05/10/14
Dashboard: Added new developer helper that shows the RGB and HEX codes of the most prominent colours on the website homepage.
Update 24/09/14
Server: Upgraded to a more performant server.
UI: Improved the responsive behaviour for tablet devices.
Dashboard: Fixed 'Link Types' visualisation bug.
Update 19/09/14
Setup: Better handling of the URL on the setup screen. Thanks to Jack21222 for reporting it.
Update 15/09/14
Content: Copy tweaks throughout.
Dashboard: Overview screen tweaks.
Dashboard: Added developer helper to help fix any Domain Canonicalisation issue.
Update 11/09/14
Dashboard Bug: Fixed visualisation.
Update 09/09/14
Homepage: Main Banner Height Fix
Update 07/09/14
Dashboard: Refined output of audit overview screen
Score: Factored in Domain Canonicalization
Update 06/09/14
Dashboard: Created new audit overview screen
Score: Captured additional information used in score generation
Update 04/09/14
Dashboard: Added recrawl button to admin navigation
Update 02/09/14
Registration: Users are now automatically logged-in after completing the registration process.
Crawl: Previous audits can now be re-run on demand
Crawl: Proxy service is only now used for external urls
Update 30/08/14
Crawl: Fixed bug to do with saving the crawl every 50 pages.
Crawl: Capturing status code information about external links.
Update 29/08/14
Setup: Can now choose whether to check external links to see if they work
Crawl: Checks external links, then as a fallback, checks the domain
Dashboard: New report on broken external links
Dashboard: New report on broken external domains
Update 25/08/14
Crawl: Fixed bug that sometimes occurred for anonymous crawls
Setup: Improved anonymous user journey
Update 24/08/14
Status: Better error flagging and a few misc. bugs.
Dashboard: Now reporting on the robots.txt crawl delay
Crawl: Fixed bug that prevented single page sites producing a valid report.
Crawl: Fixed ability to cancel crawls
Setup: Estimated time remaining for an audit is now shown on the progress modal window
Update 20/08/14
Dashboard: Added Report for captured strong / em / b / i tags
Dashboard: Added Report for captured object / svg / canvas tags
Dashboard: Added Report for Deprecated tags inc. applet / basefont / center / dir / font / isindex / menu / s / strike / u
Update 19/08/14
Dashboard: Favicons next to the showcase links
Update 18/08/14
Back-end: Cleaned up logs
Update 15/08/14
Dashboard: Improved UI to only show reports that are available.
Content: Updated the FAQ
Dashboard: When toggling between different versions of audits, the pane will remain the same for each version.
Update 14/08/14
Dashboard: Can now load previous versions of an audit
Dashboard: Misc. minor bugs thrown up in the logs
Back-end: Restructured how audit report information is stored.
Crawl: Crawl data is now saved to disc every 100 pages. This lay the foundations for pausing crawls, and restarting crawls after an error.
Back-end: Error reporting toned down.
Update 06/08/14
Crawl: Set default crawl order to be the most frequently occurring links first. This will make the page limited reports more useful.
Dashboard: Added report to show the largest images, by surface area.
Dashboard: Added report to show images with invalid values in the width and height attributes.
Performance: Added performance monitoring and metric capture to help identify performance bottlenecks.
Performance: Initial generation of report is now ~50% quicker.
Dashboard UI: Any reports that don't have any data, are hidden.
Update 05/08/14
Dashboard: Improved UI of Audit Overview page
Dashboard: Added 'Full Link Analysis' Report that shows information on URL path, query and fragment.
Update 04/08/14
Content: Added base table structure for the glossary of terms, and the SEO tool comparison analysis.
Dashboard: Added new section called 'Developer Helpers'
Dashboard: Added new 'Developer Helper' called '.htaccess Redirects'
Dashboard: Added new 'Developer Helper' called 'VHOST Redirects'
Dashboard: Added detection for deprecated HTML tags.
Dashboard Bug: Fixed domain canonicalisation reporting bug.
Update 03/08/14
Crawl: Now capturing strong, em, b, i, object, svg, canvas tags. No analysis on them yet though.
Crawl: Added capture of Google Analytics information
Dashboard Bug: Fixed bug that sometimes occurred when clicking the which resulted in no results being returned.
Dashboard: Added analytics information on Audit Overview page
Dashboard: Added new report 'Occurrences of each analytics / tracking scripts'
Crawl: Added lookup tables
Content: Updated the About Us page content.
Update 02/08/14
Crawl Bug: URLs that were entered as a start point, that then redirected to different sub-domain / domains wasn't working properly. Now it is.
Update 01/08/14
Dashboard Bug: Fixed bug that resulted in inaccurate page speed and size values being reported due to inclusion of non-status code 200 pages.
Dashboard: Added additional helper copy.
Dashboard: Added "Redirect Speed" report.
Dashboard: Tweaked the layout on the Performance reports.
Dashboard: Added anonymous audit user sign-up encouragement.
Dashboard: Colour coding on the tables to signify scale of the issue.
Crawl: Created facility to capture extra html tags for analysis.
Update 26/07/14
Setup: Added more information to the "in progress" modal window so that users know where in the process the audit is.
Dashboard: Admin Navigation - Previous and next links to cycle through the dashboards. Plan is to make this available to users.
Dashboard: Admin Navigation - Make an audit public.
Dashboard: Added loading animation when users click the 'see more' links.
Update 23/07/14
Content: Updated the Privacy Policy with cookies information, and added the TOS.
Update 20/07/14
Leaderboard: Capture score data for building leaderboard
Leaderboard: Initial version of the Leaderboard page
Setup: Added additional information to the modal window
Update 19/07/14
Leaderboard: Creation of base tables to support a leadboard
Update 18/07/14
Dashboard: Bug fixed that was introduced as a result of upgrading to the latest version of Bootstrap relating to the progress bar.
Update 13/07/14
Crawl: Captured additional attributes from <img> tags.
Dashboard: Added image information to the report.
Dashboard: Added unique image report.
Dashboard: Added most common image report.
Dashboard: Added image that don't have any 'alt' text set report.
Dashboard: Added images with 'title' attribute set report.
Update 12/07/14
Dashboard: Added extra error detection for broken crawls.
Dashboard: Started adding the image analysis.
Update 08/07/14
Upgraded Bootstrap from 3.1.1 to 3.2.0
Update 06/07/14
Dashboard UI: Added a pre-page loader, so that when large reports are loading, users know what is going on.
Update 04/07/14
Markup: Added rel="nofollow" to all of the login / registration links on the site, and excluded them from robots.txt to prevent issues with some bots.
Status: Anon crawls can now be delete en-mass
Tracking: Pane clicks on the dashboard and now captured in Piwik
Update 29/06/14
Status: Cleaned up admin options menu.
Update 27/06/14
Added server monitoring software.
Update 23/06/14
Crawls now go via a proxy service.
Update 21/06/14
Started improving, and enforcing a structure on the JS
Update 21/06/14
Performance: Reduced the page size of the dashboard by dynamically calling the keyword data.
Dashboard: Made the site URL clickable, and sorted out the CSS index issues on the pane headers.
Update 18/06/14
Content: Published a Recommended SEO Tools page.
Update 14/06/14
Dashboard: Links to the dashboard now support passing the url via the query e.g. https://metaforensics.io/dashboard?site-url=http://zoocha.com/
Dashboard: Better handling of errors for dashboard loading and retrieval
Homepage: Added public crawl screenshots to the site homepage, and tweaked the layout
Bug fix: Removed debug code from the sign-up process
Status: Take DB back-ups
Content: Added a report example sitemap page
Dashboard: Added "full screen" links next to all of the reports
Update 08/06/14
Crawl: Added function to count the number of words on a page
Crawl: Added function to count the number of sentence's on a page
Crawl: Added function to calculate the reading time of a page
Crawl: Added function to calculate the content Gunning Fog Index of a page
Crawl: Added function to calculate the content Flesch Reading Ease, and the Flesch–Kincaid Grade Level of a page
Crawl: Added function to calculate the content Coleman–Liau Index of a page
Crawl: Added function to calculate the content SMOG grade of a page
Crawl: Added function to calculate the content Automated Readability Index of a page
Crawl: Added function to calculate the content Dale–Chall readability of a page
Crawl: Added function to calculate the content Spache Readability Formula of a page
Dashboard: New report to show all of the content related information.
Update 07/06/14
Crawl: Extended the page details table to support additional content data capture
Update 06/06/14
Content: Added an FAQ section
Bug fix: Multiple values with the anchor 'rel' attribute were being analysed incorrectly.
Bug fix: Sometimes linking pages was not returning any results.
Update 04/06/14
Dashboard: Improved toggles on showing / hiding table data
Update 31/05/14
Setup: Only allow 'analyse sitemap.xml' when selecting a full crawl
Status: Added email domain link to users view
Update 30/05/14
Bug fix: Links starting with 'tel:' and 'skype:' were resolving as a 404
Bug fix: Relative links on sub-directory pages were sometimes resolving incorrectly
Update 28/05/14
Analytics: Upgraded to Google Universal Analytics, and added several click data capture points so that user behaviour can be better understood.
Update 26/05/14
Dashboard: Re-factored keyword analysis code, and extended it to include any number of keyword chains. Currently exposed is: singles, pairs, trios & fours.
Content: Added screenshots to the SEO Analysis page.
Content: Misc. copy tweaks throughout.
Update 24/05/14
Dashboard: Dynamic visualisations of keyword data, just click the next to the report heading.
Dashboard: All visualisations can now be exported in PNG, JPEG, SVG or PDF format.
Homepage: Dynamic testimonials.
Update 22/05/14
Content: Added a Privacy Policy.
Update 20/05/14
Dashboard: New view to report on just keywords
Dashboard: Added report on internal links with rel="nofollow"
Dashboard: Added report on internal links with target="_blank"
Dashboard: Calculated ratio of internal & external follow/nofollow
Dashboard UI: Introduced Pie Charts to the Audit Overview page
Crawl: Upgraded the anchor table to capture whether a path is absolute or relative
Upgraded jQuery to 1.11.1, and Parallax to the latest version.
Dashboard: Download table data that was previously not downloadable as CSV e.g. "Keywords in Anchor Text"
Update 18/05/14
Status: Favicon changes with the progress of the crawl
Dashboard: Integrated HighCharts for the charting / graphing visualisations, after ruling out ChartJS (no label option on pi charts!), Google Charts (reliant on web service), ChartSpree (usage limitations)
Dashboard: Added site audit visualisation functionality.
Dashboard: Added basic visualisation for HTTP Response Status Codes & Link Types
Dashboard: Added basic visualisation for Link Types
Crawl: Upgraded the capacity of various fields, and captured additional link anchor information including the 'rel' and 'target' values
Update 17/05/14
Crawl: Fixed bug that caused bypassing of validation checks on the 301/302 links for 'redirect_url' component
Dashboard: Added a keyword density column on all of the keyword reports
Update 15/05/14
Dashboard: Can now cycle through the different screen resolutions on the audit overview screen
Dashboard: On the duplicates report, you can now see what urls have the duplicate's
Dashboard: Updated the Score UI to include a website thumb, and exciting sliding bar charts!
Setup: Better validation on the url field.
Update 14/05/14
Dashboard Bug: Fixed download reports for the duplicate reports.
Update 13/05/14
Dashboard UI: Show the download and see all links more appropriately
Update 11/05/14
Status: Can now regenerate all public cache dashboards so that they show the latest analysis.
Dashboard: New report - keywords in anchor text
Dashboard: New report - keyword pairs in anchor text
Dashboard: New report - keywords in meta descriptions
Dashboard: New report - keyword pairs in meta descriptions
Dashboard: New report - keywords in meta keywords
Dashboard: New report - keyword pairs in meta keywords
Dashboard UI: LHN sub-navigation is now fixed to make it easier to navigate large reports
Update 10/05/14
Bug: When clicking "see all" on the dashboard, the options disappeared.
Bug: LHN on dashboard. Long title were being truncated on low screen resolutions.
Dashboard: Put the most useful reports to the top of each section.
Update 30/04/14
Bug: 301 redirect detection fixed
Dashboard: Improved 301 / 302 reporting
Update 28/04/14
Dashboard UI: Large tables now scroll horizontally.
Update 27/04/14
Dashboard: New report - keyword pairs in titles.
Dashboard: New report - keyword pairs in headings.
Update 21/04/14
Users: Can now toggle whether or not they want to see the 'tips' on the user preferences page.
Setup: Overhauled the UI on the setup screen to make it easier to use.
Dashboard: Tweaked the UX so that menu contents are better arranged.
Update 20/04/14
Core: Added stop words to the analysis.
Core: Logging system to capture and interrogate information on a per-site basis, rather than centrally.
Back-end: Logging of how long each report takes to generate.
Dashboard: New report - Keywords in headings
Dashboard: New report - Keywords in titles
Dashboard: Can now deep link to top level reports
UI: For Dashboard LHN. Enabled closing menus, and toggle arrow.
UI: Parallax is back involved on the homepage.
Update 19/04/14
Dashboard: Can now regenerate thumbnails
Backend: Integrated Composer to handle dependences
Core: Advanced URL parsing to accurately determine subdomain, TLD domain, and URL suffix.
Dashboard: New report for External suffix analysis
Dashboard: All reports now show 15 results by default, and have "see all" toggles.
Dashboard: Added new tips.
Update 17/04/14
Dashboard: Improved UI of HTTP headers block
Update 16/04/14
Status: Can now clear website screenshot cache
Score: Basic calculations for all areas
Dashboard: External links by domain
Dashboard: External links by domain (follow)
Dashboard: External links by domain (nofollow)
Dashboard: Loading spinner while generating thumbnails, plus UI tweaks for score
Update 13/04/14
Score: Grade letter generation
Score: Dynamic text for score reporting
Dashboard: Shuffle UI to make score the default landing page
Update 06/04/14
Dashboard: Detect links with underscores
Dashboard: Detect links with common language extensions
Dashboard: Calculated URL length metrics
Update 05/04/14
UI: Tweaked mark-up throughout to make better use of Bootstrap capability.
Dashboard: Detect links with spaces
Dashboard: Detect links with bad case
Update 04/04/14
Score: Improved UI for score page, and integrated into Dashboard navigation.
Update 03/04/14
Score: Rendered output of scoring algorithm as bar graph.
Update 01/04/14
Score: Finished algorithm framework for calculating the score.
Update 31/04/14
Score: Finished core work on the website Scoring / Grading
Update 14/03/14
Dashboard: Added result counts to the tables.
Update 13/03/14
Debug & Variables tables created per crawl, to store extra information.
Dashboard: Display of crawl log
Update 11/03/14
Status: Admins can now in-line edit user fields.
Update 24/02/14
Users can now register using a Twitter Account.
Sign-up & login process has been improved, including welcome email.
Dashboard: Content tweaks and additions throughout.
Dashboard: Improved heading analysis.
Update 23/02/14
Dashboard: Added report on queued links.
Homepage: Fixed Parallax issue on Chrome.
Upgraded Bootstrap from 3.1.0 to 3.1.1
Upgraded jQuery to 1.11.0
Update 19/02/14
Dashboard: Added HTTP headers. This means you can now detect the WebServer type, along with Server OS, amongst other things.
Dashboard: Updated layout of audit overview, and improved the rendering of the DNS and WHOIS information.
Dashboard: Added basic Sitemap XML insight.
Dashboard: Increased breadth of WHOIS server list.
Update 13/02/14
Can now restrict crawls by url length.
Can now restrict crawls by directory depth.
Update 11/02/14
Added new data scraping options for extracting content from lists.
Update 09/02/14
Can now deep link to sub-reports on the reports.
Update 04/02/14
Parallax eye candy for the homepage
Update 02/02/14
Upgraded Bootstrap from 3.0.3 to 3.1.0
Added Social sharing buttons to the site footer.
Update 01/02/14
Added Headroom.js to the app.
Update 29/01/14
User sessions can now be preserved with "Remember me" functionality.
Update 26/01/14
Created a logo using the Squarespace Logo tool.
Dashboard: Pages with missing Favicons
Dashboard: Content ratio
Update 12/01/14
Open Graph tags have been added to the dashboard / report page.
Update 09/01/14
CSS and JS consolidation.
Update 05/01/14
Added data scraping info page
Update 04/01/14
Users can now run a 10 page SEO audit without signing up to the application. Click the "try now" button on the homepage under the sign-up box.
Fixed various Chrome browser bugs
Added missing map file for jQuery 1.10.2
Update 03/01/14
Added Google reCAPTCHA to the app, and exposed on the contact form.
Contact form: Remember state on failed submission.
Forgotten password: Fixed bug introduced in upgrading to Twitter Bootstrap 3.0.3
Added a smattering of Glyphicons throughout.
Update 22/12/13
Dashboard: New URL length report.
Dashboard: Bug fix relating to "see all" and download report.
Dashboard: Added screenshots of website homepage taken at different dimensions.
Status screen improvements for cache regeneration.
Update 17/12/13
Small tweaks to admin options / menu
Update 16/12/13
Tweaked wording throughout so it's less "crawl" & "spider", and more "audit" & "fetch".
Upgraded to Twitter Bootstrap 3.0.3
Update 15/12/13
Dashboard: All reports are now downloadable in CSV format.
Update 11/12/13
Dashboard: Added Basic Report containing page title, meta description, meta keywords, first H1.
Setup: Fixed bug relating to the sliders passing the correct values in to the form.
Update 10/12/13
Optimised initial loading time of Dashboard for a new crawl.
Fixed setup screen UI bug for example data selections, and test data crawls.
Initial content for SEO Analysis, Link Analysis, Site Architecture
Update 08/12/13
Dashboard LHN UI bug fix
Added dashboard reports for: All Titles, Missing Titles, All Meta Tags.
Added Glyphicon's to Dashboard LHN
Improved status screen with cache detection, rebuild cache and more user friendly default values
Update 01/12/13
Dashboard page now has a left hand navigation menu
Update 30/11/13
Updated Twitter Bootstrap from 2.3.2 to 3.0.2
Update 27/11/13
Output example now merges data from different crawls
Output example now has a cache lifetime of 1 hour
Added crawl schedule information to the Groupon / Wowcher merge example
Update 23/11/13
Dashboard bug fix for "see all data"
Extended remote triggering of crawls to include by site ID and url
Update 20/11/13
Crawls can now be triggered remotely
Update 17/11/13
Fixed bug that causes the crawl to hang when calculating the content ratio
Sortable fields on the crawl setup
Tests if no data is available for the data scrape example
Upgraded to jQuery v1.10.2
Update 11/11/13
Phased out simple_html_dom as it can't handle pages with huge numbers of links.
Fixed bug due to some DB connections not being closed
Fixed the data view, for scraped data analysis
Fixed Google Analytics reporting + added goal tracking for GA & Piwik
Now have a global exclusion list of urls / domains to be ignored
Update 04/11/13
Status screen options for deleting caches etc.
Update 04/11/13
Each site now has its own cookie file, which fixes a reported shared cookie bug.
Status Screen: Filtering & paging on all_crawls, and the debug messages.
Tightened up the crawling rules around the limits per domain, per day.
Filtering fix as reported on a Reddit thread.
Suppressed error message when a site doesn't have a robots.txt file.
Fixed issue for 'multiple page scrapes' when there is in fact only one page on the site.
Sitemap.xml file added.
Recommendations - basic implementation added.
Dashboard - converted navigation to a mega menu.
Update 03/11/13
Consolidated ajax functions.
Updated Daily Deal Consolidation Example, and added to the site navigation.
Added more user insight on the status screen.
Update 31/10/13
Increased crawl limit for free users to 250.
Added Piwik analytics.
Capture user IP on registration to detect potential spam.
Added About page.
Crawl complete email notifications.
Update 29/10/13
Added forgotten password functionality.
Added more robots.txt rules.
Validation on url field on new crawls.
Update 28/10/13
Beefed up the security around the ajax requests on the status page.
Admin functionality - delete users and newsletter signups.
Registration form - fixed type "email" that some users were reporting failures on.
Update 27/10/13
Added robots.txt
Web Analytics only applied for unknown IP's.
Fixed bug in the HTML parser that was causing some crawls to error out. is_object() detection on str_get_html.
Fixed a couple of bugs reported via Reddit.
Update 26/10/13
Added a Sitemap.
Meta Descriptions & Open Graph tags added to the site.
Newsletter signup.
Update 25/10/13
You can now follow us on Twitter.
Add Google Analytics & Webmaster Tools.
Status page - added user table.
Status page - make public / private toggle.
404 error page.
.htaccess file + crossdomain.xml rules.
Fixed cancelling crawls.
Update 24/10/13
Extended registration process to prevent duplicate user names, and email addresses.
Back button now works when navigating the tabs on the dashboard page.
Crawl progress bar now works properly.
Update 23/10/13
Extended user table to support additional fields.
Implemented user settings / preferences screen.
Added test for unverified and blocked users.
Email validation for new users.
Update 22/10/13
Created user library back-end code.
Added email notifications.
Added close icon to alert notifications.
Dashboard compatibility fix for new PHP version.
Additional server and environment settings detection code.
Update 12/10/13
Added back-end D3 integration.
Update 11/10/13
Updated jQuery library.
Tweaked UI.
Version 1 - 08/10/13
Launch.