An Image Web-Service for Documentarians

The Pain

I’ve been writing docs for the Neos project, watching for ways to make it easier for more people to get involved. The Neos Editor Guide (a user manual I’m writing) really needs a lot of screenshots to be effective for the end-users that need it. But, creating docs with visuals is annoying at best, even for a technical writer like me. Screenshots are annoying to produce, and keeping them up-to-date is painful. I want to create something to make this part of documentation easier to produce and maintain.

As you write the docs, you need to create screenshots. You could interrupt the flow of writing to go create the screenshot, and then struggle to get back into the flow of writing. Or, you could try and make notes on what screenshots are needed and deal with making them later. Hopefully, you won't miss any of the image placeholders in the docs. Whichever workflow you choose, switching contexts like that causes a lot of mental friction. Taking screenshots is annoying. Plus, it’s hard to get someone else to create the screenshots for you because they have to have both sets of tools as well.

And then, someone’s got to maintain the docs. How many times have you seen manuals where the screenshots are 5 or 10 years old? Once you write a doc with lots of screenshots, no one will ever update it; If someone does update it, they won’t update the screenshots. Screenshot maintenance is tedious, so no one does it.

The Seed of an Idea

What if we automated screenshots? There are a variety of tools that we could use to do this, including Behat, Cucumber, phantomjs, ffmpeg, etc. Then, you could describe the required screenshot in the doc you are writing and let Sphinx call the other tools to generate the screenshots. However, automating image creation will create a lot of images. If you keep your docs in a git repository, like I do, your git repository will quickly bloat with all those images; that kind of bloat makes git fetches slower and makes turns the git experience into an annoyance.

So, what if all of those screenshots were stored in some external service like Dropbox or Google Drive, or some other CDN(-like) service? Then, Sphinx (or whatever doc generator you use) would need a way to get the URI of those images without hard-coding them in the docs.

A Solution

So, I'm building a web-service for documentation visuals like screenshots. The service will take an image description (sent by Sphinx or some other doc generator like Jekyll, WikiMedia, or Confluence), and return an image URI. In the background it can let users upload the screenshots or automatically create them with something like Behat. Let's say goodbye to user manuals with screenshots that are out-of-date.

While writing the docs, the author just includes a description of the screenshot. Then, as far as the author is concerned, the description of the screenshot gets replaced with an actual screenshot. No more context switching between writing and taking screenshots. Someone else or some other program can deal with actually taking the screenshots.

Imagine a website like crowdin where people upload screenshots instead of localizations. The website would list the screenshot descriptions and let users upload a screenshot for each one. Automate what we can, and leave the rest to volunteers. That gives more people a way to contribute to the docs.

Thanks!

The TYPO3 Association funded the initial development of these ideas. Check out TYPO3 CMS at typo3.org. To all of the TYPO3 Association members who have supported this effort: Thank You!

A generic screenshot webservice for documentation

@mathias.schreiber gave me some great feeback on Slack about the tools I want to build as part of my T3A budget (see my most recent blog post):

we plan to move this away from rst anyways, thus I ask myself how we benefit from the toolset

He also clarified that he's only talking about finding a tool that's easier to use, like Confluence, and it would only be for the manuals, not the reference documentation. And, he also tried to make it very clear that they aren't changing document formats right now, and there hasn't even been any official conversations in the ML. There are only plans, so far.

I'm really glad he pointed out this possibility on the horizon. Now, I need to generalize any tools I make so that they can be used by Confluence, Sphinx, or (almost) any other documentation tool that serves the needs of the project. Knowing it is on the horizon, I can design the tools to work in that context.

Here's my first stab at generalizing the screenshot tools.

Instead of starting with screenshot generation, I'm going to start with a simple web service that allows some kind of input (just an image id for the MVP, and later, it will accept a full behat scenario to describe the image that should be generated) and return a final image URI. (initially the image will serve up some placeholder/transparent image that is set to expire very quickly, and later the placeholder can be replaced with the generated image) This way, frequently regenerating screenshots in user manuals will not cause the git repo to bloat significantly. This will allow us to put the images directly in some kind of CDN service (an actual CDN if needed, or something like Dropbox or maybe Google Drive).

For the first MVP of this web service (which will only be accessible to a clients that are whitelisted, such as build.docs.typo3.org.), I will still manually generate any screenshots/screencasts. In the next phase, I will integrate behat (or something similar) that can describe the screenshots that should be generated. I'll add the ability to annotate screenshots later on.

This way, a description of the necessary screenshots can be embedded in manuals without requiring the document author to generate the screenshot (thus breaking the flow of writing the document). Then that description gets sent to the web service where something (or someone) takes the description and generates the required screenshot and makes it available at the URI provided by the web service. Even without automating the screenshot, this will allow the author, or other contributors, to go through the queue and generate all of the needed screenshots with the annotations.
This service would then need a sphinx plugin, or a wikimedia plugin, or a confluence plugin, or whatever tool is used to write manuals.

So, my budget can be useful to both TYPO3 and Neos, even if one or both teams decides to move the user manuals away from restructured text format.

Roar: Docs! Docs! Docs!

Update: I've responded to some great technical feedback on how my budget can help TYPO3 here.

This welcome conversation caught my attention on Twitter this week. I couldn’t get my comments to fit within 140 characters, so I’m responding here. Here are the tweets (without the twitter handles):

• @sorenbryder: (tweet)
  “Documentation & Documentation Tools for Neos/Flow” for 2015 is EUR 13,750. At present, 0% of this budget have been used." Come on! #neoscms
• @AskeErtmann: (tweet)This is a budget for @cognifloyd who wanted to spent time working on that now, but didn't start due to the budget situation..
• @sorenbryder: (tweet)Okay, sounds good then. Better documentation is really needed, but this isn't news to anyone :)
• @kdambekalns: (tweet)@sorenbryder And even better: The #TYPO3 documentation will probably profit from his work as well!
• @mattLefaux: (tweet)go on
• @kdambekalns: (tweet)Go [on] with… the work? Or explaining what it is about?
• @mattLefaux: (tweet)if it’s a joint venture, the budget situation can easily be resolved. Need roar info
• @kdambekalns: (tweet)So let's roar on :) The tooling (see the budget application…) could benefit all docs

I, @cognifloyd, am the budget owner for the “Documentation and Documentation Tools for Neos/Flow” 2015 budget. This is my roar! (check out the discussion of my budget application on forge. I'll just assume that you've read it in the rest of what I have to say)

As @kdambekalns said, I think the “#TYPO3 documentation will [...] profit from [my] work as well!”. Yes, my budget has “Neos” written all over it, but writing documentation alone is not enough. Dig a little deeper, and you’ll see how my work will benefit both projects. The key budget milestones are milestones 3 and 4:

• “Infrastructure\Tools for [...] documentation contributions”, and
• “Automated screenshot & video capture system for embedding in [rst docs]”.

I want to create a layer of glue between Sphinx (software that docs.typo3.org uses to render ReStructured Text, or rst) and screenshot/video capture tools like phantomjs and ffmpeg. Some people embed behat scenarios in rst, so that might be how I accomplish this. Others have already figured out how to make each of the other elements work. I want to bring the power of automated screenshots into the hands of the people that end up maintaining the docs.

When I applied for the budget, the Neos/Flow docs were ideal candidates for developing these tools. I was working on docs.typo3.org’s rendering infrastructure (more on this below), so I planned on writing those tools within that infrastructure. As a technical writer, I figured I’d be my own “domain expert” as I create the tools that I think will make it easier to create and maintain the docs. However, to be the “domain expert”, I can’t create the tools without actually writing some documentation. I need to write to identify exactly where writing/maintaining docs is painful. I need to write to make sure that the tools I create fit into the writing workflow as seemlessly as possible.

In fact, I’d started writing the Neos Editor Guide long before I applied for this budget. With this budget, I hope to make it easier for me and others to contribute to documentation efforts. It would take me a lot longer to identify gaps in TYPO3 docs than it takes to just continue what I’m already working on. Switching docs now would be like telling someone they have to rewrite all of their unit/functional tests before they can continue developing their software. Talk about lost productivity. The Neos docs are my “tests” (or at least my first test project), but TYPO3 will still get the benefit of the tools I create (or more appropriately, the tools I glue together).

I hope that TYPO3 Association members decide to continue funding my budget, even though the TYPO3 and Neos communities are separating into two overlapping communities. Both communities will benefit from better documentation tools and infrastructure. Plus, I will be able to more efficiently build these doc tools if I can continue building them as I write the docs I have already been working on.

“0% of this budget [has] been used.” (@sorenbryder)

Why haven’t I used that budget yet? Besides several personal time constraints, whenever I had time to work on something from a T3A budget this year, I prioritized building the docs infrastructure over writing/revising any of the docs. I started working on the docs infrastructure last year, and I wanted to finish that infrastructure project before I spent much time on my budget.

There has been a long-running project to replace the shell scripts that build the docs on docs.typo3.org with a web application that will make managing that rendering process easier. Any tools I write as part of my budget will be designed to plugin to this new rendering process. So, I thought working with the docs team first was the most logical course of action.

What was my budget for? Why did I apply for it?

Frankly, I applied because “docs”, for a lot of people, is a four-letter word. As @sorenbryder said, the need for better docs “isn’t news to anyone”.

But why? Why are docs always so out-of-date? Why don’t more people help keep them updated? Sure, English isn’t PHP (Sorry devs!), it’s not Esperanto, and, for most people, it’s not their native language. Writing good docs is hard work. It’s time consuming, and it requires a different mindset then writing good code does. Writing docs doesn’t feel profitable like selling another website gig, it’s not as enjoyable as a good YouTube playlist, and, when you finally get around to writing the docs, getting all the documentation tools together can be a nightmare.

There are a lot of gaps in the Neos and Flow docs. Given enough time, I could probably write new docs and revise the old docs, but that isn’t good enough for me. If I just “get it done”, the manuals I write would end up just as out-of-date as TYPO3’s Modern Template Building manual. No one has taken the time to substantially update that guide since it was written. Isn’t there a way to write a manual so that it’s easier for others to update it?

I’m a geek (ie I like PHP), English is my native language, and I’m a technical writer. Heck, I enjoy writing! I’m one of few people I know of in the TYPO3 community (and now the overlapping Neos community) with qualifications like that. Does that make me a “domain expert” when it comes to improving the documentation workflow? Maybe. I’m scratching an itch, and trying to do it in a way that will help as many people as possible.

So, I applied for a budget to help me find ways to make documentation easier for more people. But, how could I find the rough spots in writing/revising documentation without actually writing/revising a manual in the process?

That’s why I applied for my budget. There just had to be a better way!

What do screenshots have to do with that?

One of the most annoying, focus-stealing tasks in generating user documentation was generating the screenshots and short screencasts that I wanted to embed in the docs. Because they’re such a pain, and because no one takes the time to stop what they’re doing and regenerate them, I decided to automate that process. In searching the web, I discovered a variety of resources that make automating that a conceptually simple thing to do. I just need to glue the pieces together so that they can use descriptions embedded in rst docs to generate those screenshots/screencasts. If I could come up with a way to make that part of the documentaiton process better, then surely, I can find other process improvements too.

Another problem with screenshots is, everyone has their own way to take them. If numbers/labels are required, then they are going to look very inconsistent throughout a manual that gets updated by different people over the course of many years. If those screenshots are automatically generated instead, then our documentation will not only be more easy to update, but it’ll have more professional graphics as well.

What about the community split?

But, now, I’m faced with a dilemma. I didn’t know the TYPO3 and Neos communities were going to split. I think the split is good, as it is relieving a lot of tensions that have been building up in our community for many years. But, it sure makes it difficult for me to continue working on what I think will benefit both projects.

I have been in the TYPO3 community for over a decade, since version 3.4 or so of TYPO3. Back then, I was using TYPO3 to build a website for my high school. Since then, I’ve used TYPO3 and Neos in a variety of hobby and school projects. As a technical writer, I built a documentation website for one company first in TYPO3 CMS and then transitioned to using Neos, or TYPO3 Phoenix as it was called at the time. Sadly, none of the sites I’ve built still exist, but I don’t need those projects to consider myself a member of both the TYPO3 community and the Neos community.

Like I said at the beginning, I think my work will benefit both projects. My only problem was that I didn’t think a community split would happen this year. So, I made my budget mention Neos/Flow too many times instead of thinking/writing about how the TYPO3 project as a whole could benefit from these tools.

Conclusion/Request

If you are a TYPO3 Association member, please vote to honor the “Documentation & Documentation Tools for Neos/Flow” budget. I will make sure that screenshot automation and any other tools I create are smoothly integrated into the docs.typo3.org infrastructure. I’m already working hard on maintaining that infrastructure, and I look forwarding to making these features available to the docs that get rendered on docs.typo3.org.

Software Patents = Crazy Talk

Rep. Goodlatte, my representative in congress, has been working on fixing the problems with US Patent Law. I just emailed him the following:

Thank you for working on reforming our blighted patent system. Merely controlling litigation is not enough. You mentioned frivolous and speculative patent lawsuits, but you did not identify the deeper problem with software patents.

Software is written. Software should be protected under copyright laws not patent laws. Copyrights protect exact wording, but do not prevent people from speaking, thinking or writing about those ideas. Just like literate people can read ideas in good books, literate software engineers can read ideas in software. Just as literate people should not copy a book word for word and take credit for writing it, software engineers should not copy a computer program word for word and take credit for writing it.

Software should not be patentable. Think about these: A bookbinding, a new way to make paper, a new computer case, and a revolutionary computer screen. These can be patented, but the ideas on the pages of a book and the ideas in each line of software code are only thoughts and ideas. Patenting an idea is like trying to control how people think, it's like trying to control the newspapers saying that there can only be one republican newspaper and one democratic newspaper. Preventing the spread of thoughts and ideas is not constitutional.

Allowing software patents is like saying we don't really believe in the right to believe, think, or speak. Allowing software patents is fundamentally wrong. 

Please champion legislation to protect software the right way: with copyrights, not with patents.

Figuring out TDD

I want to use TDD in my GSoC project. Tests are important, and the benefits of testing are worth it.

Now, my only problem with TDD is, I'm not quite sure how to do it. Oh, I've got the mechanics in place (Travis is set up and ready to eat my github repo, and I think I've got phpunit setup locally. Heck I even have Jenkins on my local machine if I could ever get it configured right.), but whenever I go to write unit tests, I feel like I'm staring at a blank page in an old fashioned typewriter.

I feel like I'm staring at a blank page in an old fashioned typewriter. (Image from starside on flickr)

For a long time, I think my problem was an incomplete design. I knew I wanted to take stuff from a boilerplate, then "process it somehow", then put it in a derivative package. But I didn't know what that "somehow" looked like.

Well, I've figured out the main parts of that process now. I know what I want my API to look like (within PHP). So, I take my first example, and go line-by-line to make it work.

Loading Gist ....

So, let's take part of line 2:

new BlobQuery('Cognifire.SweetSitePackage');

How do I test this? Do unit tests test constructors? Is this a unit?

And so I'm back at that blank page again. I'm not quite sure what unit I'm testing. The constructor doesn't return anything, it just sets up the object. Should I add getters and setters for what should ultimately be protected properties just so that I can test that they are initialized correctly? That seems rather odd.

But TDD says don't write a line of code until you've written a failing test, so I feel stuck.

How do you approach TDD? Unit testing? What questions do you ask yourself to get past the stupid blank page?

BuilderFoundation BlobQuery

Blob: A file, or a snippet/section of a file, or syntax-tree like thing. Typically, a blob requires a special parser.

BlobType: A blob has a type so that it can be handled by the appropriate parser.

SubBlobs: A blob in another blob.

BlobType-SubType: BlobTypes are not restricted to just the main blob types, but may define subtypes as well to make working within the syntax tree easier. A SubType is separated by a dash (maybe I should use a dot as the separator like NodeTypes does in TYPO3CR. What do you think?)

Example BlobTypes include: FluidBlob, HtmlBlob, PhpBlob, YamlBlob, TypoScriptBlob. I believe the packageBuilder will need an additional DomainBlob that doesn't map to any particular media-type. A Blob is essentially a generalized syntax tree, and a DomainBlob would be used to create PhpBlobs, it's just a more specialized/focused view of the information.

Example SubTypes (if I match the syntax tree exactly):

• FluidBlob
• FluidBlob-Array
• FluidBlob-Boolean
• FluidBlob-ObjectAccessor
• FluidBlob-Numeric
• FluidBlob-Text
• FluidBlob-ViewHelper.
• PhpBlob
• PhpBlob-Class
• PhpBlob-Function (PhpBlob-Method within Class)
• PhpBlob-Interface
• PhpBlob-Namespace
• PhpBlob-Property

BlobQuery: This provides a context for FlowQuery. You create a new BlobQuery with a given path (typically a package path), and BlobQuery scans for files (optionally filtered by type), creates a Blob object for each of them, and adds them to an initial set of objects before returning a new FlowQuery instance that's using the set of Blobs.

OK, so here's a portion of the UML diagram I've been creating (in ArgoUML) to figure out what all needs to go in BuilderFoundation.

The OperationResolver expects simple values for priority, shortName, and Final, but I want to have one Operation class, BlobTypeSpecificOperations, that calls a variety of methods on the Blobs. To make this work, I add a BlobTypeSpecificOperationsAspect that calls a specialize initializeObject right after OperationResolver->initializeObject() gets called. This grabs the Cognifire.BuilderFoundation.BlobTypeOperations configuration and adds all of the registered blob operations to the list of operations. That means that if a blob has a special operation that only works for that BlobType, then add it as a method in the Blob itself (makes code organization feel better), and register that method as a BlobOperation in the Settings.yaml file. Whenever that operation gets called, it will go through BlobTypeSpecificOperations->evaluate() which will call the registered method on the blob for that operation.

The reason this is important, is you'll select a set of blobs with BlobQuery/FlowQuery, and then do some operation, and it will do that operation on all of the Blobs.

AOP is awesome!

GSoC Notebook

I wanted to show that I'm still working on this, and a video seemed like the easiest way to show off my notes to the world. Enjoy. Let me know if you have any questions or want to know about something in my notebook.