Boilerplates and Builders in TYPO3 Neos

I will edit this article to edit the In-Scope and Out-of-Scope sections as needed. I wanted to publish this sooner rather than later.

GSoC Deliverables

In-Scope for this GSoC

I plan to deliver the following packages:

• TYPO3.BuilderAbstract
• TYPO3.TemplateBuilder
• TYPO3.ZurbBoilerplate
• TYPO3.BootstrapBoilerplate

Important features / components:

• RoundTripService (very basic)
• TemplateBuilder ties into the Fluid parser
• MigrationService (very basic)
• Backend Package Manipulation
• Copy boilerplate package contents into a Deriative package
• Boilerplate Configuration in YAML
• Options layer that allows for multiple "presets" (like in TYPO3.Form). In the context of the TemplateBuilder, those "presets" should be exposed in the UI as a "theme" layer of options.
• File manipulation indicators (a way to say that you inserted a fluid widget at a particular point in a template. These will be stored in the YAML files as Eel/FlowQuery because XPath only works for XML, and we will work with more than XML)
• Fluidify SourceTemplates in TemplateBuilder (the backend for something like TemplaVoila)

Note: These are the names I prefer right now, though they've gone through several iterations. If these are not clear, please let me know. TYPO3.BuilderAbstract used to be called TYPO3.Boilerplate. TYPO3.TemplateBuilder used to be called TYPO3.FluidBuilder. SourceTemplates used to be called RawTemplates, NakedTemplates, or NakedBoilerplates.

Out-of-Scope for this GSOC

I will not have time to work much, if at all, on these:

• Anything that works with TypoScript
• Much of the UI (may be partially working thanks to TYPO3.Ice - but that's not my focus)
• Including the TemplaVoila-like UI to select the various parts of a SourceTemplate file

Package Dependencies

Boilerplates depend on the Builder that understands the files in them. Builders depend on the BuilderAbstract.
Builders extend the basic package manipulation of BuilderAbstract to handle particular filetypes
(eg TemplateBuilder handles Fluid files, and eventually will deal with SASS/LESS/CSS, etc).
BuilderAbstract also provides generic services Builder services (eg RoundTripService, MigrationService).
The PackageBuilder will probably orchestrate the other Builders when building a package, as they are complementary.
Derivative Packages never depend on the Boilerplates they are based on. They get modified copies of boilerplate files.

Producing a Derivative Package

This shows how Derivative Packages are made with the TemplateBuilder. The user has several choices:
(1) Selects a SourceTemplate (HTML/CSS) that will be converted to Fluid Templates in a Derivative Package
(2) Copy Fluid Templates from a Boilerplate Package and customize in a Derivative Package
(3) Start with nothing, and generate a Fluid Templates in a Derivative Package,
but insert some Fluid Widgets from a Boilerplate Package

I produced these images in yED. I can provide the original .graphml files if requested.

Basic Designs for TYPO3.Boilerplate

I'm designing TYPO3.Boilerplate.

Basics of TYPO3.Boilerplate

• Boilerplate Package[s] (Alternative name: Boilerplate[s]): A package that resides in FlowRoot/Packages/Boilerplates/ which has stuff that may be integrated into another package.
• Derivative Package[s] (Alternative name: Derivative[s] or Boilerplate Derivative[s]): A package that integrates one or more boilerplate packages.
• Naked Boilerplate[s]: The raw materials that makes a boilerplate without the metadata that TYPO3.Boilerplate needs to create Derivative Packages. These are often part of other packages or libraries. A site package might store the designer provided HTML/CSS files somewhere in the Resources.

TYPO3.Boilerplate is not very useful (from a user's perspective) on it's own, but is powerful within a Builder app such as FluidBuilder or PackageBuilder.

Boilerplates and Derivatives

TYPO3.Boilerplate is concerned with copying stuff from a Boilerplate package to a Derivative Package including interfaces to help Builders process or manipulate the files that get copied. Though the builders work with files, they should be able to treat them as objects via TYPO3.Boilerplate APIs.

Though I could say that Boilerplates and Derivatives have a parent-child relationship, I thought that would suggest a much more dependent or a hierarchical relationship. Derivative Packages should have a copy of everything they need from the Boilerplate. These Derivatives should never depend on Boilerplates. Only Builders that use a particular Boilerplate package should depend on the Boilerplates.

Are there situations where something other than a Builder (or a Kickstarter) can depend on a Boilerplate? One possible situation would be an agency (or maybe even Neos) that has a collection of widgets that they store in a Boilerplate package, and reuse those widgets, without modification, in their site packages. In that case, I suppose they could introduce a dependency in their composer.json, but then they run the risk of upgrading one little thing in their boilerplate only to break things in the dependent site packages. It would be better to copy the widgets into the site packages, and then have a Boilerplate Migration that helps to update site packages from one Boilerplate version to the next. In any case, I don't think there should a technical limitation that prevents the dependency; It'd be one of those only-do-this-if-you-know-what-you're-doing kind of things.

I wonder if TYPO3.Neos.NodeTypes could be considered a Boilerplate package.

Please respond in the comments, or in the Neos forum, and let me know what you think of this design direction.

Naked Boilerplates and Boilerplate Metadata

In initial designs, I used the term Virtual Boilerplate, but I think Naked Boilerplate is a more appropriate term, because it has templates, but they are missing any boilerplate metadata. It is up to a Builder Package whether or not they support Naked Boilerplates. For some Builders, like PackageBuilder, it won't make much sense. For others, like FluidBuilder (TemplateBuilder?), it makes a lot of sense to support templates that have not been converted into Fluid templates already.

These HTML/CSS/JS templates that have not been broken into Fluid templates are Naked Boilerplates. For example, Twitter Bootstrap is a Naked Boilerplate. To create a Fluid Boilerplate Package, Twitter Bootstrap gets integrated into some Fluid templates with metadata that tells TYPO3.Boilerplate

• which Builder to use (FluidBuilder) and
• what stuff is available in the Boilerplate (lists of layouts, widgets, options like changing colors, etc)

The Builder uses that metadata to present a list of stuff to the user. They can then integrate that stuff into their own Derivative Package. FluidBuilder could create fluid templates in an Application package or a Site package using the widgets/options (stuff) that the user chooses from the list of available stuff.

The site package that is based on Twitter Bootstrap does not have information about all of the widgets available in the Boilerplate package; Instead, it contains metadata about which parts of the Boilerplates were integrated into it (e.g. which widgets were included in the site or application package). Ideally, the Derivative Package could have enough information to facilitate upgrades from one Boilerplate version to the next (deltas or diffs or steps taken to create the site package from the old boilerplate. These changes would be rebased--like git--on the new version to help with template upgrades).

Derivative Metadata

Derivative Packages will need metadata about:

• Integrated Boilerplates
• Which Boilerplates have been integrated
• What stuff (layouts/widgets/components) came from a particular Boilerplate (I assume that if it didn't come from the Boilerplate, then it was added by the developer, so care should be taken not to overwrite those changes.)
• How things were integrated (like where widgets were inserted) could use Eel/FlowQuery (my preference) or XPath (which only works on XML-like files, but could be very useful in some cases.)
• Versioning Information to facilitate Migrations
• Kickstarters and Builders might not support migrating from one version of a Boilerplate to the next, but they should store something that says which version of a Boilerplate was used in generating the derivative package.
• Neos and other agencies might provide a Boilerplate package with a collection of widgets that can be added to templates. How/where these widgets are inserted should be carefully recorded, so that changes may be replayed/rebased on any manual template modifications or to facilitate migrating a package from one boilerplate version to the next.
• Builder-specific options
• For example, the FluidBuilder (maybe TemplateBuilder would be a better name) would have a theme layer that uses theming options. If a Boilerplate can be customized with custom colors (think SASS or LESS), then the Builder could expose those options in some kind of interface.

Initial Thoughts on Code

Note: I need to go through Flow's code again to decide which parts can be reused. Possibilities include:

• Configuration (Store all the metadata in yaml files)
• Monitor (File Monitor / Change Detection, Possibly implement ChangeDetectionStrategyInterface to look for specific changes in target files)
• Package (This has some cool stuff, but I can't depend on a package being "activated" before I interact with it, since Builders and Kickstarters will be creating packages. Packages might be broken until a Kickstarter/Builder is finished creating it.) 
• Reflection (Possibly only useful in builders that work with PHP...)
• Resource (Could be very helpful, as long as it's not restricted to just the Resources Folder. This needs to be able to interact with any file in a package)
• Utilities (File manipulation stuff)

For the services and the domain (which does not get persisted to a database, everything is stored in files), we have:

• Model
• Packages/ (Builders should extend these to provide the business-logic of how to manipulate relevant parts of a Derivative based on metadata from the Boilerplate)
• AbstractPackage implements PackageInterface
• BoilerplatePackage extends AbstractPackage
• DerivativePackage extends AbstractPackage (might be a Site or Applicaiton)
• (Others like library could be implemented at some point if someone needs it, but they might not extend the PackageInterface... I'm not sure on that, and I think it is outside the scope of this project.)
• NakedBoilerplateInterface
• Factories (extended in Builder packages to provide implementation details of how to work with different parts of a package)
• Copy between packages (Boilerplate to Site; Boilerplate to Application)
• Repository (Each package type has a separate repository that extends the basic repository)
• BoilerplateRepository is the collection of BoilerplatePackages
• SitePackagesRepository is a collection of DerivativePackages
• ApplicationRepository is a collection of DerivativePackages
• Service
• RoundTrip service (like PackageBuilder: Detecting changes in watched packages to trigger updates in registered Builders -- Uses FileMonitor)
• Migration service (assists in migrating a Derivative from one Boilerplate version to the next, assuming the boilerplate included migration scripts. For inspiration: Doctrine Migrations, TYPO3CR Migrations, Flow Code Migrations, git rebase and friends)

Boilerplate Math

Calculus is my inspiration for and explanation of how TYPO3.Boilerplate works. I call this, Boilerplate Math.

NOTE: I don't think people need to understand that I got the terms from Calculus to use the Boilerplate API. "Integrate" and "derive" are common enough words, that I think the meaning will be easy enough to understood for anyone who needs to use the Boilerplate API. But, for those that might want to modify Boilerplate, the history of them might come in handy.

Consider these basic calculus equations:

Integrate f'(x) to get f(x) plus some change

Derive f'(x) from f(x) plus some change

If I say that f'(x) is a Fluid Boilerplate package that gets integrated into f(x), a SitePackage, then:

Integrate Fluid (f) Boilerplate (B) into SitePackage (P) with Changeset (C)

Derive Fluid (f) Boilerplate (B) Package (p) from SitePackage (P) with Chagneset (C)  

A SitePackage might use parts from more than one Fluid Boilerplate. For example, it could integrate some widgets from twitter bootstrap as well as widgets from an agency's custom boilerplate that they use for many of their websites.

Designing TYPO3.Boilerplate

So, I've been thinking about boilerplates in TYPO3 Flow + Neos (it's my GSOC project). After filling up a couple of whiteboards, I started writing stuff in my notebook. I don't expect my notes to mean very much to anyone but me, but I love to see how other people work, so I'm posting all of my notes so that you can see how I work.

edit: I removed the iframe because it was demanding more attention than it deserved. Check the notes out here.

Flow Package Types + Boilerplate Thoughts

So, there are a bunch of different package types for TYPO3 Flow. If you look in the packages folder you'll see several major kinds:

• Distribution/ : The super-package that includes (via composer.json) all the other packages
• Packages/
• Applications: Flow-based apps
• Framework: Flow's packages
• Libraries: 3rd party libs and non-Flow packages
• Sites: Neos site packages
• Boilerplates: Meta-templates (part of my GSOC Proposal) that contain boilerplate files (php code, fluid templates, yaml, ...)

Then, there are some common naming conventions for Flow applications.

• FooKickstarter: for initial creation of packages of type "Foo".
• FooBuilder: to not only create, but edit "Foo" packages.

So, if my GSOC project is only going to be a one-time copy kind of a package, then I need to call it FluidKickstarter. If I give it the ability to edit and not just start templates, like I plan, then it should be called FluidBuilder.

I'm thinking of creating a Boilerplate framework. That way, there'd be a TYPO3.FluidBoilerplate that deals with Fluid specific template manipulation, but the TYPO3.Boilerplate package would contain the more abstract parts of dealing with any Boilerplate package. So that would mean there'd be something like this:

• FooBoilerplate:
• extends TYPO3.Boilerplate to assist builders in building "Foo" packages.
• Not to be confused with packages in the Boilerplates folder, which contain raw material for the Boilerplate system to use in creating various types of packages (by FluidBuilder, PackageBuilder, FormBuilder, etc)

That solution feels awkward. Perhaps, there's a TYPO3.Boilerplate package, like before, that provides the abstract ways of working with Boilerplate packages. Then, the Builder packages would include a server-side layer that interacts with TYPO3.Boilerplate to do the building of packages (Fluid, Package (Flow), Form, Ember). Plus, each Builder package would contain a Frontend app that does the same thing as the backend, but in the browser. Then they would communicate over some RESTful API with Ember Data.

To date, that has been the solution for the other builder projects, so I think that would be a good convention to follow, with one change, in that I'll be using Boilerplate package, and hopefully other Builders will follow suite.

One of the issues with current Builders is that they store all of the template like stuff in their Private Resources folder. That's not very extensible, and though it might work for a lot of the current Builders, it won't work for FluidBuilder because I want there to be a lot of ready-made boilerplates that people can use to create whatever site they want as fast as they want. They should be able to make their own, or use Twitter Bootstrap, or H5BP, or whatever other grid system they enjoy.

Why? Flow is already fun for developers, but it needs to attract designers. So, Neos should be fun for designers to use, and Boilerplate packages are a good way to get there.

Ice and PackageBuilder and Boilerplates, Oh my!

So, I was just browsing github, and stumbled on TYPO3.Ice (ICE=Interface Construction Environment). Looking around, there's also a project on forge, and a git.typo3.org repo. I wanted to make sure it wasn't a previous effort to build TYPO3.FluidBuilder (my GSOC proposal), so I asked Rens on IRC what Ice was all about.

TYPO3.PackageBuilder

Apparently, it's part of the effort to create a TYPO3.PackageBuilder (see the github project, the forge project, and the git.typo3.org repo). PackageBuilder is like the next generation of the ExtensionBuilder that we know and love from TYPO3 CMS. It'll have a domain building UI, so that you can graphically create your domain objects and link them all together. I'm very excited to see someone working on the PackageBuilder.

TYPO3.Ice in TYPO3.PackageBuilder

PackageBuilder needs a UI similar to TYPO3.FormBuilder, so some of that interface was abstracted out, so that they could be reused in multiple packages. That common UI lives in TYPO3.Ice.

TYPO3.FluidBuilder

The best part of that? I'll probably be able to use TYPO3.Ice when I create TYPO3.FluidBuilder for a GSOC project. Maybe it'll be possible for me to create the UI portion in FluidBuilder after all, since it's mostly built for me already.

Boilerplates in TYPO3.PackageBuilder

Plus, looking through PackageBuilder's code, it looks like PackageBuilder could use the Boilerplate concept as well. Right now, there's Resources/Private/CodeTemplates. They've got both TYPO3 CMS and Flow templates, which could be put in separate boilerplate packages, so that you don't have to update the whole PackageBuilder package every time one of the boilerplates gets updated. Plus, then other organizations could create their own Flow Package Boilerplates, and use PackageBuilder to create packages with them.

Resources/Private/Generator could become Boilerplate packages in SiteKickstarter and Kickstart. Resources/Private/NewFormTemplates in FormBuilder could also be a Boilerplate, though one yaml file might not warrant it, so it could be treated like a virtual boilerplate. Potentially, other people could create Form boilerplates, though.

I'm very glad that I found out about Ice and PackageBuilder. It looks like my GSOC project will be very useful indeed (who wants their work to die and rot unused in the backroom?!). I hope that the Boilerplate concept, and the FluidBuilder will help more people to start using Flow and get involved with Neos.

Neos (dev-master) Hints

I'm using Neos from dev-master. Here are some things I've run up against with my group.

Workspaces

You can see and work in anyone's workspace, not just your own. If you want to work in the same workspace with someone, just edit the url.

@ in a Neos URL tells you what workspace you're looking at or using. For example, this shows that I'm in cognifloyd's workspace (hey, that's me!).

If I wanted to see the workspace for user john316 I would replace "@user-cognifloyd" with "@user-john316". Eventually, there will be some kind of access control to say which users can interact with which workspaces, but for now, once you're logged in, everything's fair game.

One way to use this is to share a workspace with a group of colleagues. Before I found this, we would each publish our changes (move them live) every time we wanted to share our work. This way, we can all work in the same workspace, and publish it live when everyone has finished their part.

Wireframe Mode

Resizing images in wireframe mode is currently broken.

Sections don't show up reliably until you've got content in them.

When you aren't in wireframe mode, you can just drag-n-drop elements to re-organize them. However, this feature isn't in wireframe mode. Instead, click "Inspect" in the lower left hand corner. You can rearrange the elements in there.

When you add a page, be sure to edit the nodepath in the right hand properties bar or you'll get some ugly urls. Neos doesn't seem to be automatically creating the nodepath based on the page name yet.