Mar
25
Dark Clouds Overhead?

Dark Clouds Overhead?

The "Cloud."

Ever since a bunch of celebrities found themselves sprayed all over the internet without their permission, there has been a lot of fear and misunderstanding about the cloud.

What is the "cloud?" Where did it come from? Where is it? How does it work?

Of course, as with anything technology related, we could go into eye-glazing detail about how the cloud works and what the various layers are and do.

But for most of us, think of these Big 5 companies:

Apple, Google, Microsoft, Amazon, Facebook.

Where do you "go" when you use their services? When you store photos, videos, files in their software, where is it?

In actual fact, it is copied to several bunkers (think Fort Knox), with armed guards and big fences. Each of those bunkers (data centres) consumes the same amount of electricity as a medium sized town, mostly to keep the computers inside from overheating.

Those big 5 (along with a few other enterprise companies like Oracle and Salesforce) buy more high end PC parts together than all the consumers in the world do, and some of them build their own PCs, ordering parts straight from the factory. They all use custom software to manage all of those files.

You can't get into the data centre unless you have a real reason to be there. No tourists.

They are connected with big fat cables of fibre optics to the internet, and due to their size and the amount of traffic that flows through them, the internet is adjusted to take them into account.

But the physical location of files (copied on multiple hard drives in multiple locations so that any one failure won't lose your precious photos) is the least of our concerns.

Back in the day, when us computer engineers were designing a system, there was always a boundary between what we controlled and what we didn't control. If we were drawing the system on a whiteboard, there was always a "cloud" off to the left, or at the top, indicating the source of things that we didn't control. Stuff in the "cloud" just happened, and we reacted to it.

Many, many companies have decided that they were going to become that "cloud." And the more they compete with each other, the more tools that are created to do stuff for us, so that our area of concern can shrink down to just a few things.

In fact, one can make a business run using only cloud tools, connecting tools together like toy building blocks. And in fact, there are several businesses focused on allowing you to do that!

In future posts, I will discuss cryptography and security, what to worry about and what not to worry about, and how to relax instead of worry when dealing with the cloud.

Look for a North Creek Flow Coaching session, starting in the Edmonton area soon!

Cheers,

Dave
@NorthCreekSoft

PS #DataButler is adding more cloud capability in its DNA and so the release date has been moved out.

Categories: Business , Software

Feb
06
No SQL to be found

No SQL to be found

 

SQL: Structured Query Language. The technology used by conventional databases (Oracle, MySQL) to store data in rows and columns. Experts in SQL command a high price in the marketplace as Database Administrators (DBAs) or programmers.

NoSQL: Not Only SQL, or Non-SQL databases. A big trend in IT, these databases use views of data other than rows and columns to store and organize information. They may use trees and paths (like a file system on a computer) or documents (like a filing cabinet with indexes).

DataButler is designed for people who have data, but don't have the resources within their business to get into SQL or NoSQL. They are non-SQL shops, and there are millions of them. You may be one of them. You have some accounting application, a bunch of spreadsheets, a folder full of word processing templates that you use as forms. You may have an app or two on your smartphone that simplifies your workload. You may even have a group of SaaS products that allow you to run your business online without anything more than a web browser running on your laptop.

But data is one of the drivers of your business (that and money). If you don't know who, what, where, when, and how, you are making bad decisions and losing money.

Hiring a person to organize all of your data and track the KPIs and metrics is unrealistic. Figuring out how to integrate all the different apps out there could take another full time employee. Let's make this easier.

DataButler is designed for people who know spreadsheets. Spreadsheets are simple, extensible, and valuable for non-SQL people. But the value in databases is the connections they allow, and the connections they uncover. DataButler understands spreadsheets, sees your data, and can make connections (over time, between different spreadsheets, between different authors).

You might not need SQL or NoSQL yet. But you do need Data. In fact, you have Data, but it's not being used effectively yet. DataButler will help you get there.

Sign up for the beta here.

Categories: Software

Feb
05
Small Data

Small Data

Everyone is excited about Big Data. Millions of data sources, streaming full time into vast data stores, tended by giant-brained data scientists, producing "insight" for mega-businesses.

But you need to worry about payroll, A/R, A/P, inventory, cash - maybe a few other things. Nothing too big, and you probably have a KPI* that you watch most closely of all.

Not Big. Small Data.

The 90s produced the spreadsheet, a great tool for small data. You can what-if, you can dream, you can look at scenarios and goal-seek. Spreadsheets can even do Medium Data, but mostly you just do Small Data, the important stuff.

If you don't need Big Data, but you do need better visibility into your Small Data, DataButler will help by organizing it behind the scenes. When you need information from someone, DataButler will create a simple form and a link you can send, and it will integrate the information you get back into your spreadsheet.

If you are tracking several things (locations, people, projects), if they have a unique ID, you can put that ID in column A in a bunch of spreadsheets and DataButler will link them all together into one big list of data, notifying you if there are different cell values in that row from different sources.

DataButler will help clean up your spreadsheets by noticing what is changing and what is staying the same over time.

A Butler isn't a scientist. A Butler is personal. A Butler deals with the little things.

Let us help you with your Small Data.

Sign up for the Beta at northcreek.ca.

*Key Performance Indicator

Categories: Software

Jan
23
Launch Countdown

10...9...8...

 

We are getting close to the end of January. DataButler is not quite ready to launch yet, but we do have a signup for people who want to be in on the beta. We will start admitting people to the beta based on (a) who signs up first, and (b) who invites the most friends to the party. Please help to get the word out!

The product vision is becoming clearer - the repository is the base product (what I called CXL). But integration within the repository will be a huge part of the value - just by giving a row a unique ID, you'll have the equivalent of a join in a database, with a merge feature that will collect all the rows that have the same ID.

ListMaker will also be a huge value add within DataButler - instant, one-touch web forms based on the columns in the spreadsheet that you need filled in. Lots of people have quick web forms - these will tie back into your spreadsheet immediately. And you can use a full spreadsheet as a base for a ListMaker form - just designate some columns and the new data will be added to the document as it comes in.

There are lots of places where you could use this - how many times have you texted someone with some detail questions? Now you can text them the hyperlink to a form. They fill out the form, hit submit, and the answers are in your document.

Screenshots and demos coming soon!

Sign up here!

Categories: Business , Software

Jan
06
Data Flows

 

Data Flows

Businesses exist as a series of data and material flows, human decisions, and relationships. North Creek is aiming to improve your data flows. That can give you better insight into your business, and help you make better decisions. It may even get you home earlier, improving your relationships!

Primary data needs to be gathered - from POS terminals, from workers in the field, from surveys. That data gains value as team members look at it, sort it, collate and conflate it, and make expert judgments based on it.

That data becomes the source for more judgment calls and more analysis. BI tools offer a way to drill down from charts all the way back to "primary" data. But that primary data is just what is available to the tool.

It makes sense that spreadsheets are used as the analysis and sorting tool most of the time. Whether online or on your desktop, putting data into rows and columns that can be shifted around and manipulated makes sense to knowledge workers.

The problem is that spreadsheets are a dead end - hard to import data in, hard to export knowledge out - unless they are in the hands of experts.

North Creek wants to make the flow in and out easier. Mobile data collection that goes straight into a master spreadsheet. Linking of related data across spreadsheets within a corporate repository. The ability to drill "back" to the primary data that led to the numbers you are seeing. This will be possible.

Help guide North Creek's product vision. Email me, or contact me via Facebook or Twitter. My LinkedIn account also links back to this blog.

In 2015, let your business data flow.

Categories: Business , Software

Dec
18
Christmas break - the Beta is getting closer

 

North Creek is going on vacation for the next week. If I have time, I will send out some tweets and update my Facebook page, but this blog will likely be pretty quiet until the New Year.

In the new year, North Creek will be launching two products - Listmaker and CXL. Branding for these products may change as we go through Beta. A quick summary:

1) Listmaker:

This is going to be a free app that will let you collect data from your teammates / friends / meetup participants / whatever. You upload a spreadsheet with an empty table in it - no data, just column headers. We convert that table into a simple web form. We give you a URL - something like

https://northcreek.ca/listmaker/7468db85-2d8f-466d-abdf-063ffd707030 (Just a sample, not live)

and a QR code that points to the same URL.

You share those links however you like. Anyone who follows those links gets access to an empty form where they can provide the data for your spreadsheet. You can get notified when anyone or a certain number of users fill in the form.

When you come back to northcreek.ca, you log in (using an email address or some public identity (Facebook / Twitter / LinkedIn / Yahoo / Microsoft) and you get to see all the forms that you have in our system. As the editor of the form, you are the only one who gets to see the data others have provided.

When you are satisfied that you have collected enough data (you have enough for a pot-luck, or you have the whole team's contact info), you close the collection and download the spreadsheet, with all the data filled in for you.

This will be ad-supported, with options to pay to remove ads and add different styles to the default look of the form.

2) CXL:

This will be our flagship application. CXL will take fully-featured spreadsheets and create a "document" record around them. Within that document record, you will be able to assign visibility and editing rights to people based on their North Creek identity (which again, can be based on email or on another public identity provider). You will be able to track changes to the spreadsheet by row, column, formula, format, or data.

CXL will give you a preview of the data and the charts in your spreadsheet, as of the latest version, or of any version in the document history. You will be able to make minor edits to the spreadsheet from the web interface, but primarily, the web page will serve as a window into the repository that surrounds the spreadsheet. If someone wants to "fork" your spreadsheet, they can do so and retain the document history up to the fork.

CXL will be compatible with .xls and .xlsx records to start, with other file formats becoming available in the future.

The goal will be to create an understanding of a spreadsheet and to secure it as a useful, reliable part of your business process. To that end, North Creek will work on integrating other data sources into your document so that you can create a template and fill it with data using web services or other cloud data providers. The outputs will always include a real spreadsheet document, but other outputs could be created as well.

In future releases, CXL could extract a template out of your spreadsheet, removing volatile data. You could also upload a template and associate it with a document record. The outcome of that could be the creation of a community resource of spreadsheet templates, with the ability to share useful spreadsheet designs with your team, your company, or the world.

I think spreadsheets are under-utilized because IT can't get a handle on them. CXL will allow you to see and manage your spreadsheet inventory, removing IT's concerns and solving your problems at the same time.

CXL will start with basic functionality in January. Listmaker will launch when ready. Stay tuned!

Contact: Email, Twitter, Facebook, LinkedIn, or this blog.

Categories: Business , Software

Dec
16
Productivity - and first steps to community

Productivity

Some days you can just get into the "flow" of things and stuff just gets done. Some days you have to push harder to get going. Usually, once I get started, with my tools set up and even one small thing accomplished, I can keep the momentum going and crank some real output.

I was told (it might have been in "The Pragmatic Programmer" by Hunt and Thomas) to leave one easy task undone at the end of every day, so you'd be starting on a downhill slope the next morning.

Are your tools set up for that?

If your business runs on a bunch of disconnected files and folders, and you have to remember where you are and what the next step is every morning, you may have a difficult time getting into a "flow state". On the other hand, if you are in the real world, and your tools are designed to fit some idealized view of how business works, you may be spending your time trying to force your tools to fit your reality.

I love the freedom of starting with an empty spreadsheet when I'm trying to imagine something. You can just create a table over here, a formula over there, and then a chart just because. I might be a bit of a spreadsheet geek, or maybe I haven't spent enough 50-hour weeks staring at them. I can always go back to my programming environment or the command line. But spreadsheets are only useful once you've done the setup work - setting them up is a cost.

What if someone else has a template that will get you halfway there? What if we can share solutions (not data, but formulas, chart designs, table arrangements)?

Open source software has bloomed through sites like SourceForge and GitHub. North Creek sees the potential for a smaller, but still global exchange of spreadsheet development effort. CXL is the first step to making that happen.

Nobody is going to share their corporate data with the world. But if your spreadsheet is already in the CXL repository, and it has been analyzed and compartmentalized using North Creek's software, a template could be extracted out of your working spreadsheet that could help others.

What do you gain? Membership in a community that recognizes your contribution, and access to great ideas from others. The sense that you're not the only one dealing with this mess.

First steps - you could be part of creating something new. Sign up by emailing me, "like" North Creek Software on Facebook or follow me on Twitter, or comment on the blog (within the blog using Disqus, or using the ShareThis buttons up top). Do you work with spreadsheets? Are you a whiz at something? Would it be nice to have a library of templates to share, so that you would be halfway there and going downhill when you start tomorrow morning?

Categories: Business , Software

Dec
13
Integration via Spreadsheet?

 

In many ways, businesses use Excel as an edge node in the graph of integration pathways. Data comes in to your spreadsheet, is folded, mutilated and spindled, and turned into a table or chart you can put into your slide deck.

We all know Excel and its competitors can do more than that, but for all the reasons I've described previously, few businesses choose to go that way. It's too easy to change a spreadsheet - it's one big sandbox for playing with data! So businesses lock things down with custom software or integration tools.

If we had a repository that tracked and provided "adult supervision" for spreadsheets, could we use spreadsheets closer to the middle of the integration graph? Could we trust some spreadsheets as tools to provide some visibility into the guts of your data as it moves through your business? Instead of being stuck in a database that only a DBA "Wizard" can peer into, would it be helpful to be able to see data as it is being transformed?

Excel and others are pretty good at subscribing to data feeds (using ODBC or other connectivity, or through CSV file imports). Part of North Creek's technology stack is Apache POI, which is a library that takes apart a spreadsheet like a chicken, allowing me to see its guts and play with each individual cell. CXL was envisioned as a repository with intelligence, allowing businesses to trust that their spreadsheets were not being changed and distributed without the document owner's knowledge. What if there was the ability to put data feeds into CXL, not just spreadsheets?

I am not planning on re-inventing any wheels. I would embed Mule as a front end and require some configuration on your part to set up the data feeds. But any data source can be expressed as an html page (I am using jmesa for now) or as a spreadsheet table. This could be interesting.

If you want to help steer North Creek's product vision, sign up for the beta by emailing me at dave@northcreek.ca.

You can also like my Facebook page or follow me on Twitter. If you are interested in receiving these blog posts via email, tell me in an email!

Categories: Business , Methodology , Software

Dec
12
Good Spreadsheets vs. Bad Spreadsheets

Spreadsheet "Design"

Spreadsheets are usually "grown", not designed. If you are responsible for a spreadsheet that is vital to a business process, relied upon by others - you may not be able to describe the parts of the spreadsheet. It just is. There are the places you go to update the data. There are the changes you make to the dates. There are the parts that always break and the parts that never change.

So what makes a good spreadsheet?

Let me know in the comments or on social media what you think. I have an opinion, but there are experts out there!

Spreadsheets are a container for data and programming code. So my experience as a database administrator and designer, and as a programmer across multiple languages (and eras) suggests that there may be some patterns that would be useful.

1) Separation of Concerns.

Many programmers use the shorthand SoC for this, and it is one of the pillars of good design in most coding schools of thought. Make every component of your [thing] do one thing, and that one thing well. For spreadsheets, that probably means looking at the structure of your worksheets. If you have everything on one worksheet, it might make things easier to move some of the content onto another sheet and use relative links to bring the data back for calculation purposes.

A rule of thumb - separate the spreadsheets into sections based on how fast things change. If you have parts that never change, put them in a worksheet and don't touch that worksheet. If it changes, you should know why. If you have stuff that changes all the time, isolate that from the rest of the document so that the volatility doesn't lead to accidentally overwriting some of the longer-lived stuff. If you have stuff that changes on a schedule, work on a naming scheme and put all of the date-sensitive stuff on one worksheet. Then you can turn that into a template and change only that stuff on a schedule.

2) Don't Repeat Yourself

This comes from "The Pragmatic Programmer" by Hunt and Thomas (right up there on my bookshelf). If you enter data twice, one of the times will be wrong - not every time, but often enough that there will be a problem at some time in the future. So try to avoid the cut and paste, and especially the re-typing. If you can automate the export and import of data from your other tools, do it! The fewer the steps involved, the better.

There are plenty more - let me know your best ideas, or the source of your best Spreadsheet tips!

...Follow me on Twitter @NorthCreekSoft - Like North Creek Software on Facebook - email me at dave@northcreek.ca...

Categories: Methodology , Software

Dec
11
Spreadsheets and Databases

 

Custom Spreadsheets.

It's obvious to the database administrator when he (usually he - ask me sometime about one awesome female DBA I worked with) hears those words that this is going to be a mess.

Someone who doesn't know much about computers (or they wouldn't have done this on a spreadsheet) has poured hours and hours into a set of worksheets that work "just right." All the formulas are just so, the graphs are exactly what we want to see, and the tables can be sorted and filtered to a fare-thee-well.

Until it doesn't work.

Someone adds a row in the wrong place. Data is improperly typed in. The wrong format data is imported. Someone forgets to change the "Month" cell that a bunch of careful analysis is based on.

Suddenly, it's IT's problem. Or - if you are a small business owner - it's your problem, and even if you created the spreadsheet, it's not an easy solution.

Where's the backup? There used to be formulas in this column - what were they supposed to be? How was this supposed to be updated at the end of the month?

If North Creek is going to solve the "Spreadsheet Problem," we need to hear all of these stories. The gnarly, painful, exhausting work of untangling a spreadsheet that fit like my old pair of slippers, until suddenly it didn't. Let me know in the comments. Like "North Creek Software" on Facebook. Follow @NorthCreekSoft on twitter. Find me (David Block) on LinkedIn. Whatever - I want to hear your horror stories.

I also want to connect with you experts that know the tools to get us out of this mess. I will not have the answer to every problem - despite my optimistic marketing claims. I am going to solve one big piece of it.

How is that going to work?

You have to think in multiple dimensions - I'm going to start with 5(!) and we'll see how far we get.

1) Spreadsheet Data
Spreadsheets are tables, and we love putting data in table form. All SQL databases work off of tables. CXL will extract the data from the spreadsheet you upload, so that we can track changes to it as the document evolves.

2) Spreadsheet Programming
The difference between a Word table and an Excel data table is that in Excel, you can do things to the data! You can use formulas (formulae if you are pedantic) to do calculations. You can use VBA to do things even Excel can't do. You can then display the results in informative graphs and charts. Everything in the document is stored in the file (xml if you are using .xlsx), so it can all be extracted and stored. CXL will isolate the programming parts of the document as much as possible, so that we know what's editable in what way. Some times (model generation) the data is static and the calculations are volatile. Usually, the calculations change relatively slowly, and the data underneath changes.

3) Cell formatting
Cells are little documents, and each has a format. Many times, the formats will be consistent down a column, or across a row. But the differences are meaningful. Spreadsheets are meant to have a displayed component somewhere. That needs to be tracked. This is where .csv export and import fail.

4) Rows
One of the simple pleasures of working with a spreadsheet is the moment when you realize you've forgotten a row, and you just insert it in the right place in the middle of things. As long as it doesn't break that absolute reference you put in that other place... Rows are often inserted or deleted, and tracking that change by individual cell (a) is tedious, and (b) misses a piece of information. The user was inserting or deleting a data record of some kind (usually), or adding a calculation that wasn't there before. This could be noticed as a change.

5) Columns
Of course, anything you can do to rows, you can do to columns. Again, it may be a whole new record, or more likely, a whole new attribute across all the records, that you are adding or removing. New calculations across the entire data set. Storing that change as a bunch of cell additions turns that useful information into a bunch of noise.

CXL is aiming to be a smart, spreadsheet-savvy document repository. Sharepoint and other document management systems can store versioned spreadsheets and keep the "latest" copy available. CXL will break your spreadsheet down across at least these five dimensions, tracking each change where it makes the most sense. The evolution of your document becomes a story CXL tells back to you. Of course there will be security and user roles and easy access. But CXL will see into your document, helping you develop it and protecting you from that change that necessitates that call to some expensive "expert" who will tell you to quit using spreadsheets and get a real application already.

I think we can make your spreadsheet into a part of your business that you don't have to worry about.

dave@northcreek.ca 

P.S. The Beta will start in January. We are looking for a few more businesses who have spreadsheet problems that need fixing. Let me know if you are one of them!

Categories: Business , Software

Dec
10
Spreadsheets and the failure of IT

 

When I was a corporate programmer, spreadsheets were evil.

There were too many of them, they contained important and valuable data, and we (the IT people) couldn't keep them under control.

People would have them on USB sticks, on Dropbox, on their kid's laptop. And the calculations were stuck in an obfuscated binary format, hard to get at and hard to reuse.

So why are spreadsheets still around?

1) They are easy. Nothing to a good model or a good project breakdown in 20 minutes. No other tool gives you that much flexibility.

2) They are flexible. Add a column. Add a row. Change the formula on the third linked worksheet and look at the graph change.

3) They are personal. You can do things on your desktop (or laptop) (or tablet) that others don't have to see or laugh at you about. You can spitball a scenario. You can what-if. You can write one version of the project summary that is not for publication, then actually erase it and write a not-so-candid version.

So where do spreadsheets live?

Wherever change is happening.

A new company doesn't have the budget or the insight to commit to a database schema that won't change. It can't know whether this tool or that tool will match its process. So spreadsheets work during growth.

During disruption, trying to fit the new reality into an ERP package built for the last decade (and designed during the decade before that) leads to frustration and error. A spreadsheet tracks the data as things change around you.

CXL will use a gentle touch. Keep your spreadsheet. Keep your formulas. We'll find the data in the spreadsheet. We'll track changes as they happen, whether they are within a cell, a new row, a new column. The history will be there. And you will be able to manage change.

CXL will do branching. Spreadsheets are personal, and the other person's template may be a good starting point for them, but not for you. Or maybe you want to take a model in an entirely different direction. Branch the document, don't lose the history, and continue on your own path.

CXL will be easy. Just a file upload, and you have a document in a secure repository. There will be a public URL if you want to share with the world. There will be private access tokens that you can hand out. There will be the option to share only with a certain group of people, that have to log in using their email address. And for each level, you can restrict to preview only, download privileges, the right to propose changes, or document ownership. You can make things complicated if they need to be. But simple if that's all that you need as well.

Suggestions? Comments? Let me know. Email me at dave@northcreek.ca  to join the Beta in January, or to subscribe to the North Creek mailing list.

Categories: Methodology , Software

Mar
11
More Symfony, thoughts on PHP

Just finished my 4th day of the Symfony tutorial. It does pack quite a bit into a one-hour (supposedly) lesson, and if you cut'n'paste the code from the web page, things move pretty quick. But it looks like a pretty well-designed project layout, with lots of small folders so that things are put in the right place, and lots of convention-over-configuration. I'll keep at this, get through the tutorial, and maybe I'll write that music management app I've been thinking about once I get a handle on this.

I actually touched some php today. As a long-time Perl bigot, I wasn't sure I'd like it, but if you think of it as crippled, purpose-built web perl, it seems okay. Perl 6 is still not here, so I'll live with what I have access to on the server (as I said yesterday).

Now for some Magento, then I have to prepare for a real-life interruption - a ski trip, just when I was generating some momentum with this coding stuff. I'll bring the MacBookPro, and we'll see how much bandwidth (and time) I get on the mountains (Fairmont, BC).

Categories: Software

Mar
10
PHP, Symfony, Magento

Welcome back to all of you who have missed me for the past 11 months :-)

I have done a few things - my house is now liveable (in the basement), and I have a GST # and a few clients. I am now working on a couple of proposed projects - I need to spend some time up front so that I have a demo to show them.

1) Magento - a free (to start with) e-commerce php package. Setting up at http://www.northcreek.ca/store, this will be a development web store. We'll see how far Magento can go.

2) Symfony - a PHP library that helps build web apps quickly and cleanly. I want to build a music management app for Onoway Baptist Church, where I am the worship leader, and possibly soon the webmaster. Symfony gives me a good starting point.

So why PHP? Well, I have a hosted server with GoDaddy (that you are reading this on), and they support PHP, not perl, java, ruby, or others. No root access. So I'll go as far as I can with this setup, then move to a java-supported server setup when required. In the meantime, I've installed the Entropy.ch php package from Marc Lyanage (www.entropy.ch) on my laptop, so that I can develop using php and mysql at home.

I will try to keep things updated - I'm currently on Day 3 of the Symfony trial. Lots of configuration headaches due to my laptop. However, with very little description of a model, I now have a database and a website that edits the database. Hooray for modern web frameworks, even in PHP! :-)

Categories: Software

Mar
19
Registered with Elance

I spent this morning registering with Elance, an online virtual outsourcing broker. I think this is a positive step for North Creek. Most of the jobs on Elance are bite-sized, and I can build up my portfolio (from nothing) with a few well-done jobs (and develop a set of references for the business, not just for David Block, employee).
In filling out the profile, I discovered some things about myself. I am not too confident in delivering a beautiful, dynamic, functional website (using PHP and some CMS), but I am ready to compete on requirements gathering, architecture, design, data modelling, and back-end coding. So that is where I can make some money, and then as I develop a better toolset for graphical front ends (whether desktop or web), I can start competing in that space as well.
My goals are crystallizing for North Creek's architecture. I want to write nothing from scratch after a job has been won - even for the detailed customizations that every client will expect. I want to build a suite of generic applications that can be quickly modified to meet the needs of my client base. My goal is 18 hour turnaround - if you tell me in a morning meeting what you would like to see, I want to be able to provide a working demo (not a hack) by the next morning meeting. This provides me with a framework around which to make architectural choices.
In a surprise move, I am quite seriously considering learning C++. I now have two books (one on generic C++, one on building with Qt), and have downloaded the relevant frameworks from Qt that will allow me to write serious applications for multiple environments. I have spent my entire working life with multi-tier environments, but that doesn't seem to be a big factor in small businesses, so this evolution seems to make sense. I will keep this blog updated as I progress and make more choices.

Categories: Business , Software

Feb
19
Book Review: The Pragmatic Programmer

Well, I don't expect these to be comprehensive reviews, but just a few notes about the books I have finished.

I have begun a regimen where I read a chapter out of a book every day. This keeps a constant velocity, and lets me get through books that have been on my shelf for years. I have enough varying interests that I am reading 4 book chapters/day. This takes some time away from coding right now, but I am learning so much that I accept the cost. Eventually I will get disciplined enough to do my reading in the early morning, before the world awakes. I expect that will be later, though, when the log house is complete and I can stumble over to my office and sit in a comfortable chair.

The Pragmatic Programmer was written in 1999 by Dave Thomas and Andy Hunt. It has stood the test of time, mostly because it avoids too much specific technology and focuses on the principles of effective programming. I can see the influence of this book in the book I'm working on now, "Agile Software Development: Principles, Patterns, and Practices" by Robert Martin. "Pragmatic" has become a code word and has been branded as such in the industry. The "Pragmatic" authors have taken their own advice and started learning a new language every year, which is profitable for them, since they often get to write the book about said new language. They jumped on Ruby fairly early, if I recall. It makes sense: this book has a lot of Perl in it, and Perl is so 20th Century.

Metaprogramming is a big deal for them - getting scripts to write code. This has become standard in the industry, and .Net and Java annotations do this as part of their internal structure. However, that, and DRY (don't repeat yourself) are the big takeaways from this book.

They are fumbling about the agile way, but XP was just being articulated at the time, so things like Unit Testing and short iterations were still somewhat controversial. Now they are not controversial, just not done in practice. Hmmm.

Don't Repeat Yourself is a huge deal. If I could get my code to that level, I'd be ecstatic. Redundancy is a real enemy. It is the one thing that killed EJBs, I think. Spring is just so much more terse. It is also the reason I don't like Hibernate. I have a database schema that already exists; I don't want to repeat it in awkward XML. I also don't want to give a single application power over a database schema - often the data is much longer-lived than the application. So I want to work from the data out. That's what led me to the design for DBDB.

Getting good at an editor! What a good idea! I used to think in Emacs keystrokes, but I've been away from coding for long enough that the Mac way is more natural now. I don't know if that will stay, or if I'll get back to Emacs, which has so much more power available without lifting my fingers off of the keyboard. With this laptop, I have a touchpad not too far away, which makes things somewhat easier, but still, keeping the fingers on the home row is definitely a win overall.

I really appreciate the wisdom in "The Pragmatic Programmer," but I am nowhere near that level yet. I will keep working towards it - it feels like I may have to put this one in a rotation, and read it again in a year or so, to keep the target in front of me.

In the meantime, I have plenty of other books in front of me.

What did you think of "The Pragmatic Programmer?" What, you haven't read it yet? Or are you like I was, and felt so guilty about how you actually work that you never finished it?

Categories: Books , Software

Feb
18
Coding Options

I have discovered that I have been away from coding for a while! My last coding assignment was with EDS(www.eds.com, now a unit of HP), working on an existing codebase that dealt with a mixed Oracle/mainframe backend, with a Java servlet/JSP front end (http://travis.gov.ab.ca). This was a familiar environment, since I worked on similar systems at Novartis (www.gnf.org) and the University of Alberta Human Metabolome Database Project (metabolomics.ca). However, my work at EDS transitioned into Business Analysis, including a lengthy stint as a facilitator for a large-scale Smart Card implementation. While I was gone, the world changed!

The world has experimented with web user interfaces (UIs), and has found them wanting. In their place, the Rich Web has gained prominence. This primarily has meant AJAX, and in practice, the Google Web Toolkit along with other offerings. Adobe has introduced Adobe AIR, which allows cross-platform desktop development based on Flash. And the rise of the Mac platform to relevance has re-invigorated cross-platform library solutions, such as Qt, a unit of Nokia, which provides Java and C++ development environments that can be targeted at almost any platform, including Linux, Mac, Windows, and many smart phones.

So I want to develop custom, easily modified business software. So do millions (?) of other developers. I hope that my choice of technology gives me a business advantage. Of course, my primary business advantage is that I will start by creating a relationship with my clients, and then will meet their perceived and actual needs. This means that my choice of platform must allow me to fulfill those needs within a cost and time structure that allows me to be profitable.

So, what do I do? Here are some major factors in my decision-making process (which has not finished yet!):
1) Run on desktop vs. remote server: Do I set up a server and allow my clients to log in to the server? Or do I send a complete solution to the client desktop, including a persistence solution and all business logic?
My experience and my instincts suggest that if I can control the server, I can improve the experience for the client without the pain of an upgrade process. This should allow me to iterate rapidly towards a really good product.
2) Web client vs. Rich client: Should the user interface be compiled to run on the local machine, or should it be mediated by one of several possible browsers, over an uncertain network connection? This seems to be somewhat of a false dichotomy, as long as the client has decent bandwidth. With my choice of a server backend in #1, the bandwidth seems to be required. Can a solution such as GWT support enough functionality to make the browser "disappear?" Or would a Java client offer enough speed and interactivity to make the browser interface seem clunky? Perhaps I should let Qt provide the UI framework, and write once, test everywhere?
3) Client hosting of data: Will the clients be comfortable with their private business data in the "cloud," or should I sell them on local appliances that host their solution on their local intranet (which means in many cases, setting up an intranet)?
The low cost of desktop hardware leads me towards this second option, where I build and configure a server that hosts the persistent data and the application, which is served to the client desktop (through something like Java Web Start, or through GWT, or Adobe AIR). This allows me to set up backup schemes and encryption to reassure the client of the security of their data, and gives me a single upgrade point per client.

Given the explosion of virtual server environments available, I believe this progression of questions and answers will lead me to develop a standard appliance that can be easily loaded into a commodity PC. This appliance will host the applications required by the client, while the data will be stored on the PC outside of the appliance, allowing for quick swapping of images when I wish to push out an upgrade or bugfix, without touching the client data.

I have a lot of experience with MySQL, and combined with its price point (free!), I plan to stick with it for now (while watching PostGres et al). Combining MySQL with a JVM and a servlet container (Tomcat? Jetty? Jboss? I'll decide later... the joy of standards) gives me a comfortable backend. Now, developing on the front end for the JVM has gotten interesting, and that's where I'm still experimenting. GWT? AIR? I will probably pass on AIR for now, as its advantages are pretty small compared to the other options, and there is a small upfront cost involved. I'm still happiest with free tools.

I am taking a good long look at Qt, and its Java implementation, Qt Jambi. That combination gives me a near-native front end (from C++ compiled into native code) combined with Java (which I actually have worked in before). If it's not dynamic enough (more on that in a later post), I will probably invest some time in GWT, and then fall back to HTML+javascript as a last resort (but the one I actually know how to do now).

Suggestions?

Categories: Business , Methodology , Software

Feb
09
A little progress

Well, a bit of an update.

I'm coaching a basketball team (my son's Grade 7 team), which takes up some brain space on Mondays (practice) and Wednesday or Thursday (games). They are just beginners, but that's fine, so am I :-)

So today I just got started on a database schema, and then had to plan my offense. The practice went okay, but the kids need way more repetition before they do the offense naturally. So hopefully they at least have fun.

My database schema - very basic:


  mysql> show tables;
  +----------------------+
  | Tables_in_northcreek |
  +----------------------+
  | event                |
  | participant          |
  | party                |
  | time                 |
  | type                 |
  +----------------------+
  

These tables will be the start of a generic framework that allows me to create business applications using metadata rather than code. At least that's the plan. I need a template system for the UI (View), a workflow system for the Controller, and an extensible, Universal data model for the Model. My goal is 1 day turnaround from requirements to prototype. So Monday requirements meeting with a client, Wednesday demo of those requirements in a production app.

If I can get this to work, then I should be able to undercut commercial business accounting software and still make money on support contracts. But we're a ways from having a Quickbooks feature set. That's the target, though.

I may use Qt as the development framework - cross-platform development including smart phones and Java. It depends on if I can figure out how to template a dynamic UI that runs off of metadata. I know I can do that using web technologies, but that introduces servers and browsers, etc., which is not all bad, but might be overkill for a small office.

I've also downloaded Scala to play around with - the alpha geeks have given it a thumbs up, and it runs in the JVM, so I can integrate it with everything else out there (including Spring). I'll keep this blog posted - I just completed a Hello World app that actually ran in Eclipse :-)

Bedtime for the kids. Out.

Categories: Business , Software

Feb
06
Comments!

So I'm attempting to install a commenting system - hope to hear from some visitors!

We'll see if it works.

I am using blogtools, which hosts the comments and just requires me to update my templates (well, Thingamablog's templates) to pop up a comment window.

Doesn't seem to be working yet - I will update soon.

Another attempt? YES!!! I plan to always celebrate little successes like this one - maybe by breaking for lunch :-)

Categories: Software

Feb
06
Blogging with Thingamablog

I spent most of yesterday looking for a free desktop blogging client that doesn't rely on a blogging API to post. I have a hosted domain with GoDaddy (northcreek.ca) and I just want to ftp a rendered blog entry up to the host, without a lot of complicated installs (mysql, wordpress, etc.). I was using Radio Userland, but it seems to be abandonware (Dave Winer, I know you're busy, but Radio breaks my Mac!). I rebooted about 10 times yesterday, when usually I reboot only on Software Updates.

I looked around at Google's suggestions, and ended up with Thingamablog. This is a Java application that does exactly what I need. The only problem was that it was a .jar file with a bunch of associated lib jars, nothing like a Mac application. Apple has done a good job of making Java apps look "close enough" to real Mac apps that I can deal with the strange icons, etc., but I wanted a double-clickable .app bundle that I could move around.

So I made one! I got a lot of help from a bunch of websites that pointed out the basics, and I wouldn't want to rely on this app being distributed like this, but it seems to work so far.

I'll stick with this until (a) I find something better, also for free, or (b) I write my own. Of course, Thingamablog is free (GPL), so I can always pick up the source code and make it better.

I cut'n'pasted an icon from the splash screen - I am not a graphic artist! - but let's see how Thingamablog does with images:

Splash Screen:

Thingamablog Logo  

New Thingamablog Icon:

Thingamablog Icon  

And there we go!

Categories: Software

Feb
05
Introduction

Welcome to the North Creek Blog. This will be a record of the creation of a new software development company in northern Alberta.

My name is David Block. I am starting the company I would like to work for (Thanks for the idea, Joel). It will use agile, Test-Driven Development, and it will assemble Open-Source tools and libraries to create custom applications for small businesses that have the capabilities of much larger, more expensive commercial software. I plan to stand on the shoulders of many other giants, and see what comes of the process.

I have a few ideas of my own. I like working off of a real database engine, so I will start with a general database schema that will store most of the nouns in the program. Then, I will build a workflow solution on top of that schema that thinks in terms of verbs. The verbs will lead to the dynamic generation of the interface needed.

I have a library of database interaction code that I have developed over several previous projects, for different employers. The latest name for it is DBDB. I will use it as the ORM for my application. It is very lightweight, and requires very little maintenance to adapt to dynamic database schemas. I don't like maintaining a big xml file - flashbacks to J2EE.

I will also use a lot of Spring for setting up the project.

I will try to post reviews of useful books, projects, and libraries as I try them out.

Currently, I'm trying to read some of the books that have been sitting on my shelf for a while: JUnit Recipes, The Pragmatic Programmer, and Uncle Bob's Agile Software Development. Now I have to try to apply those principles as a small business owner, working on my own.

Hopefully I'll see you around.

Categories: Books , Business , Methodology , Software