Dark Clouds Overhead?
The "Cloud."
Ever since a bunch of celebrities found themselves sprayed all over the
internet without their permission, there has been a lot of fear and
misunderstanding about the cloud.
What is the "cloud?" Where did it come from? Where is it? How does it
work?
Of course, as with anything technology related, we could go into
eye-glazing detail about how the cloud works and what the various layers
are and do.
But for most of us, think of these Big 5 companies:
Apple, Google, Microsoft, Amazon, Facebook.
Where do you "go" when you use their services? When you store photos,
videos, files in their software, where is it?
In actual fact, it is copied to several bunkers (think Fort Knox), with
armed guards and big fences. Each of those bunkers (data centres)
consumes the same amount of electricity as a medium sized town, mostly
to keep the computers inside from overheating.
Those big 5 (along with a few other enterprise companies like Oracle and
Salesforce) buy more high end PC parts together than all the consumers
in the world do, and some of them build their own PCs, ordering parts
straight from the factory. They all use custom software to manage all of
those files.
You can't get into the data centre unless you have a real reason to be
there. No tourists.
They are connected with big fat cables of fibre optics to the internet,
and due to their size and the amount of traffic that flows through them,
the internet is adjusted to take them into account.
But the physical location of files (copied on multiple hard drives in
multiple locations so that any one failure won't lose your precious
photos) is the least of our concerns.
Back in the day, when us computer engineers were designing a system,
there was always a boundary between what we controlled and what we
didn't control. If we were drawing the system on a whiteboard, there was
always a "cloud" off to the left, or at the top, indicating the source
of things that we didn't control. Stuff in the "cloud" just happened,
and we reacted to it.
Many, many companies have decided that they were going to become that
"cloud." And the more they compete with each other, the more tools that
are created to do stuff for us, so that our area of concern can shrink
down to just a few things.
In fact, one can make a business run using only cloud tools, connecting
tools together like toy building blocks. And in fact, there are several
businesses focused on allowing you to
do that!
In future posts, I will discuss cryptography and security, what to worry
about and what not to worry about, and how to relax instead of worry
when dealing with the cloud.
Look for a North Creek Flow Coaching session, starting in the Edmonton
area soon!
Cheers,
Dave
@NorthCreekSoft
PS #DataButler is adding more cloud capability in its DNA and so the
release date has been moved out.
Categories:
Business
,
Software
Dark Clouds Overhead?
No SQL to be found
SQL: Structured Query Language. The technology used by conventional
databases (Oracle, MySQL) to store data in rows and columns. Experts in
SQL command a high price in the marketplace as Database Administrators
(DBAs) or programmers.
NoSQL: Not Only SQL, or Non-SQL databases. A big trend in IT, these
databases use views of data other than rows and columns to store and
organize information. They may use trees and paths (like a file system
on a computer) or documents (like a filing cabinet with indexes).
DataButler is designed for people who have data, but don't have the
resources within their business to get into SQL or NoSQL. They are non-SQL
shops, and there are millions of them. You may be one of them. You have
some accounting application, a bunch of spreadsheets, a folder full of
word processing templates that you use as forms. You may have an app or
two on your smartphone that simplifies your workload. You may even have
a group of SaaS products that allow you to run your business online
without anything more than a web browser running on your laptop.
But data is one of the drivers of your business (that and money). If you
don't know who, what, where, when, and how, you are making bad decisions
and losing money.
Hiring a person to organize all of your data and track the KPIs and
metrics is unrealistic. Figuring out how to integrate all the different
apps out there could take another full time employee. Let's make this
easier.
DataButler is designed for people who know spreadsheets. Spreadsheets
are simple, extensible, and valuable for non-SQL people. But the value
in databases is the connections they allow, and the connections they
uncover. DataButler understands spreadsheets, sees your data, and can
make connections (over time, between different spreadsheets, between
different authors).
You might not need SQL or NoSQL yet. But you do need Data. In fact, you
have Data, but it's not being used effectively yet. DataButler will help
you get there.
Sign up for
the beta here.
Categories:
Software
No SQL to be found
Small Data
Everyone is excited about Big Data. Millions of data sources, streaming
full time into vast data stores, tended by giant-brained data
scientists, producing "insight" for mega-businesses.
But you need to worry about payroll, A/R, A/P, inventory, cash - maybe a
few other things. Nothing too big, and you probably have a KPI* that you
watch most closely of all.
Not Big. Small Data.
The 90s produced the spreadsheet, a great tool for small data. You can
what-if, you can dream, you can look at scenarios and goal-seek.
Spreadsheets can even do Medium Data, but mostly you just do Small Data,
the important stuff.
If you don't need Big Data, but you do need better visibility into your
Small Data, DataButler will help by organizing it behind the scenes.
When you need information from someone, DataButler will create a simple
form and a link you can send, and it will integrate the information you
get back into your spreadsheet.
If you are tracking several things (locations, people, projects), if
they have a unique ID, you can put that ID in column A in a bunch of
spreadsheets and DataButler will link them all together into one big
list of data, notifying you if there are different cell values in that
row from different sources.
DataButler will help clean up your spreadsheets by noticing what is
changing and what is staying the same over time.
A Butler isn't a scientist. A Butler is personal. A Butler deals with
the little things.
Let us help you with your Small Data.
Sign up for the Beta at northcreek.ca.
*Key Performance Indicator
Categories:
Software
Small Data
10...9...8...
We are getting close to the end of January. DataButler is not quite
ready to launch yet, but we do have a signup
for people who want to be in on the beta. We will start admitting people
to the beta based on (a) who signs up first, and (b) who invites the
most friends to the party. Please help to get the word out!
The product vision is becoming clearer - the repository is the base
product (what I called CXL). But integration within the repository will
be a huge part of the value - just by giving a row a unique ID, you'll
have the equivalent of a join in a database, with a merge feature that
will collect all the rows that have the same ID.
ListMaker will also be a huge value add within DataButler - instant,
one-touch web forms based on the columns in the spreadsheet that you
need filled in. Lots of people have quick web forms - these will tie
back into your spreadsheet immediately. And you can use a full
spreadsheet as a base for a ListMaker form - just designate some columns
and the new data will be added to the document as it comes in.
There are lots of places where you could use this - how many times have
you texted someone with some detail questions? Now you can text them the
hyperlink to a form. They fill out the form, hit submit, and the answers
are in your document.
Screenshots and demos coming soon!
Sign up here!
Categories:
Business
,
Software
Launch Countdown
Data Flows
Businesses exist as a series of data and material flows, human
decisions, and relationships. North Creek is aiming to improve your data
flows. That can give you better insight into your business, and help you
make better decisions. It may even get you home earlier, improving your
relationships!
Primary data needs to be gathered - from POS terminals, from workers in
the field, from surveys. That data gains value as team members look at
it, sort it, collate and conflate it, and make expert judgments based on
it.
That data becomes the source for more judgment calls and more analysis.
BI tools offer a way to drill down from charts all the way back to
"primary" data. But that primary data is just what is available to the
tool.
It makes sense that spreadsheets are used as the analysis and
sorting tool most of the time. Whether online or on your desktop,
putting data into rows and columns that can be shifted around and
manipulated makes sense to knowledge workers.
The problem is that spreadsheets are a dead end - hard to import data
in, hard to export knowledge out - unless they are in the hands of
experts.
North Creek wants to make the flow in and out easier. Mobile data
collection that goes straight into a master spreadsheet. Linking of
related data across spreadsheets within a corporate repository. The
ability to drill "back" to the primary data that led to the numbers you
are seeing. This will be possible.
Help guide North Creek's product vision. Email
me, or contact me via Facebook
or Twitter.
My LinkedIn
account also links back to this blog.
In 2015, let your business data flow.
Categories:
Business
,
Software
Data Flows
North Creek is going on vacation for the next week. If I have time, I
will send out some tweets and update my Facebook page, but this blog
will likely be pretty quiet until the New Year.
In the new year, North Creek will be launching two products - Listmaker
and CXL. Branding for these products may change as we go through Beta. A
quick summary:
1) Listmaker:
This is going to be a free app that will let you collect data from your
teammates / friends / meetup participants / whatever. You upload a
spreadsheet with an empty table in it - no data, just column headers. We
convert that table into a simple web form. We give you a URL - something
like
https://northcreek.ca/listmaker/7468db85-2d8f-466d-abdf-063ffd707030
(Just a sample, not live)
and a QR code that points to the same URL.
You share those links however you like. Anyone who follows those links
gets access to an empty form where they can provide the data for your
spreadsheet. You can get notified when anyone or a certain number of
users fill in the form.
When you come back to northcreek.ca, you log in (using an email address
or some public identity (Facebook / Twitter / LinkedIn / Yahoo /
Microsoft) and you get to see all the forms that you have in our system.
As the editor of the form, you are the only one who gets to see the data
others have provided.
When you are satisfied that you have collected enough data (you have
enough for a pot-luck, or you have the whole team's contact info), you
close the collection and download the spreadsheet, with all the data
filled in for you.
This will be ad-supported, with options to pay to remove ads and add
different styles to the default look of the form.
2) CXL:
This will be our flagship application. CXL will take fully-featured
spreadsheets and create a "document" record around them. Within that
document record, you will be able to assign visibility and editing
rights to people based on their North Creek identity (which again, can
be based on email or on another public identity provider). You will be
able to track changes to the spreadsheet by row, column, formula,
format, or data.
CXL will give you a preview of the data and the charts in your
spreadsheet, as of the latest version, or of any version in the document
history. You will be able to make minor edits to the spreadsheet from
the web interface, but primarily, the web page will serve as a window
into the repository that surrounds the spreadsheet. If someone wants to
"fork" your spreadsheet, they can do so and retain the document history
up to the fork.
CXL will be compatible with .xls and .xlsx records to start, with other
file formats becoming available in the future.
The goal will be to create an understanding of a spreadsheet and to
secure it as a useful, reliable part of your business process. To that
end, North Creek will work on integrating other data sources into your
document so that you can create a template and fill it with data using
web services or other cloud data providers. The outputs will always
include a real spreadsheet document, but other outputs could be created
as well.
In future releases, CXL could extract a template out of your
spreadsheet, removing volatile data. You could also upload a template
and associate it with a document record. The outcome of that could be
the creation of a community resource of spreadsheet templates, with the
ability to share useful spreadsheet designs with your team, your
company, or the world.
I think spreadsheets are under-utilized because IT can't get a handle on
them. CXL will allow you to see and manage your spreadsheet inventory,
removing IT's concerns and solving your problems at the same time.
CXL will start with basic functionality in January. Listmaker will
launch when ready. Stay tuned!
Contact: Email, Twitter,
Facebook,
LinkedIn,
or this blog.
Categories:
Business
,
Software
Christmas break - the Beta is getting closer
Productivity
Some days you can just get into the "flow" of things and stuff just gets
done. Some days you have to push harder to get going. Usually, once I
get started, with my tools set up and even one small thing accomplished,
I can keep the momentum going and crank some real output.
I was told (it might have been in "The Pragmatic Programmer" by Hunt and
Thomas) to leave one easy task undone at the end of every day, so you'd
be starting on a downhill slope the next morning.
Are your tools set up for that?
If your business runs on a bunch of disconnected files and folders, and
you have to remember where you are and what the next step is every
morning, you may have a difficult time getting into a "flow state". On
the other hand, if you are in the real world, and your tools are
designed to fit some idealized view of how business works, you may be
spending your time trying to force your tools to fit your reality.
I love the freedom of starting with an empty spreadsheet when I'm trying
to imagine something. You can just create a table over here, a formula
over there, and then a chart just because. I might be a bit of a
spreadsheet geek, or maybe I haven't spent enough 50-hour weeks staring
at them. I can always go back to my programming environment or the
command line. But spreadsheets are only useful once you've done the
setup work - setting them up is a cost.
What if someone else has a template that will get you halfway there?
What if we can share solutions (not data, but formulas, chart designs,
table arrangements)?
Open source software has bloomed through sites like SourceForge and
GitHub. North Creek sees the potential for a smaller, but still global
exchange of spreadsheet development effort. CXL is the first step to
making that happen.
Nobody is going to share their corporate data with the world. But if
your spreadsheet is already in the CXL repository, and it has been
analyzed and compartmentalized using North Creek's software, a template
could be extracted out of your working spreadsheet that could help
others.
What do you gain? Membership in a community that recognizes your
contribution, and access to great ideas from others. The sense that
you're not the only one dealing with this mess.
First steps - you could be part of creating something new. Sign up by emailing
me, "like" North Creek Software on Facebook
or follow me on Twitter,
or comment on the blog (within the blog using Disqus, or using the
ShareThis buttons up top). Do you work with spreadsheets? Are you a whiz
at something? Would it be nice to have a library of templates to share,
so that you would be halfway there and going downhill when you start
tomorrow morning?
Categories:
Business
,
Software
Productivity - and first steps to community
In many ways, businesses use Excel as an edge node in the graph of
integration pathways. Data comes in to your spreadsheet, is folded,
mutilated and spindled, and turned into a table or chart you can put
into your slide deck.
We all know Excel and its competitors can do more than that, but for all
the reasons I've described previously, few businesses choose to go that
way. It's too easy to change a spreadsheet - it's one big sandbox for
playing with data! So businesses lock things down with custom software
or integration tools.
If we had a repository that tracked and provided "adult supervision" for
spreadsheets, could we use spreadsheets closer to the middle of the
integration graph? Could we trust some spreadsheets as tools to provide
some visibility into the guts of your data as it moves through your
business? Instead of being stuck in a database that only a DBA "Wizard"
can peer into, would it be helpful to be able to see data as it is being
transformed?
Excel and others are pretty good at subscribing to data feeds (using
ODBC or other connectivity, or through CSV file imports). Part of North
Creek's technology stack is Apache POI,
which is a library that takes apart a spreadsheet like a chicken,
allowing me to see its guts and play with each individual cell. CXL was
envisioned as a repository with intelligence, allowing businesses to
trust that their spreadsheets were not being changed and distributed
without the document owner's knowledge. What if there was the ability to
put data feeds into CXL, not just spreadsheets?
I am not planning on re-inventing any wheels. I would embed Mule
as a front end and require some configuration on your part to set up the
data feeds. But any data source can be expressed as an html page (I am
using jmesa for now) or
as a spreadsheet table. This could be interesting.
If you want to help steer North Creek's product vision, sign up for the
beta by emailing me at dave@northcreek.ca.
You can also like my Facebook
page or follow me on Twitter.
If you are interested in receiving these blog posts via email, tell me
in an email!
Categories:
Business
,
Methodology
,
Software
Integration via Spreadsheet?
Spreadsheet "Design"
Spreadsheets are usually "grown", not designed. If you are responsible
for a spreadsheet that is vital to a business process, relied upon by
others - you may not be able to describe the parts of the spreadsheet.
It just is. There are the places you go to update the data. There are
the changes you make to the dates. There are the parts that always break
and the parts that never change.
So what makes a good spreadsheet?
Let me know in the comments or on social media what you think. I have an
opinion, but there are experts out there!
Spreadsheets are a container for data and programming code. So my
experience as a database administrator and designer, and as a programmer
across multiple languages (and eras) suggests that there may be some
patterns that would be useful.
1) Separation of Concerns.
Many programmers use the shorthand SoC for this, and it is one of the
pillars of good design in most coding schools of thought. Make every
component of your [thing] do one thing, and that one thing well. For
spreadsheets, that probably means looking at the structure of your
worksheets. If you have everything on one worksheet, it might make
things easier to move some of the content onto another sheet and use
relative links to bring the data back for calculation purposes.
A rule of thumb - separate the spreadsheets into sections based on how
fast things change. If you have parts that never change, put them in a
worksheet and don't touch that worksheet. If it changes, you should know
why. If you have stuff that changes all the time, isolate that from the
rest of the document so that the volatility doesn't lead to accidentally
overwriting some of the longer-lived stuff. If you have stuff that
changes on a schedule, work on a naming scheme and put all of the
date-sensitive stuff on one worksheet. Then you can turn that into a
template and change only that stuff on a schedule.
2) Don't Repeat Yourself
This comes from "The Pragmatic Programmer" by Hunt and Thomas (right up
there on my bookshelf). If you enter data twice, one of the times will
be wrong - not every time, but often enough that there will be a problem
at some time in the future. So try to avoid the cut and paste, and
especially the re-typing. If you can automate the export and import of
data from your other tools, do it! The fewer the steps involved, the
better.
There are plenty more - let me know your best ideas, or the source of
your best Spreadsheet tips!
...Follow me on Twitter @NorthCreekSoft
- Like North Creek
Software on Facebook - email me at dave@northcreek.ca...
Categories:
Methodology
,
Software
Good Spreadsheets vs. Bad Spreadsheets
Custom Spreadsheets.
It's obvious to the database administrator when he (usually he - ask me
sometime about one awesome female DBA I worked with) hears those words
that this is going to be a mess.
Someone who doesn't know much about computers (or they wouldn't have
done this on a spreadsheet) has poured hours and hours into a set of
worksheets that work "just right." All the formulas are just so, the
graphs are exactly what we want to see, and the tables can be sorted and
filtered to a fare-thee-well.
Until it doesn't work.
Someone adds a row in the wrong place. Data is improperly typed in. The
wrong format data is imported. Someone forgets to change the "Month"
cell that a bunch of careful analysis is based on.
Suddenly, it's IT's problem. Or - if you are a small business owner -
it's your problem, and even if you created the spreadsheet, it's not an
easy solution.
Where's the backup? There used to be formulas in this column - what were
they supposed to be? How was this supposed to be updated at the end of
the month?
If North Creek is going to solve the "Spreadsheet Problem," we need to
hear all of these stories. The gnarly, painful, exhausting work of
untangling a spreadsheet that fit like my old pair of slippers, until
suddenly it didn't. Let me know in the comments. Like "North Creek
Software" on Facebook. Follow @NorthCreekSoft on twitter. Find me (David
Block) on LinkedIn. Whatever - I want to hear your horror stories.
I also want to connect with you experts that know the tools to get us
out of this mess. I will not have the answer to every problem - despite
my optimistic marketing claims. I am going to solve one big piece of it.
How is that going to work?
You have to think in multiple dimensions - I'm going to start with 5(!)
and we'll see how far we get.
1) Spreadsheet Data
Spreadsheets are tables, and we love putting data
in table form. All SQL databases work off of tables. CXL will extract
the data from the spreadsheet you upload, so that we can track changes
to it as the document evolves.
2) Spreadsheet Programming
The difference between a Word table and an
Excel data table is that in Excel, you can do things to the data! You
can use formulas (formulae if you are pedantic) to do calculations. You
can use VBA to do things even Excel can't do. You can then display the
results in informative graphs and charts. Everything in the document is
stored in the file (xml if you are using .xlsx), so it can all be
extracted and stored. CXL will isolate the programming parts of the
document as much as possible, so that we know what's editable in what
way. Some times (model generation) the data is static and the
calculations are volatile. Usually, the calculations change relatively
slowly, and the data underneath changes.
3) Cell formatting
Cells are little documents, and each has a format.
Many times, the formats will be consistent down a column, or across a
row. But the differences are meaningful. Spreadsheets are meant to have
a displayed component somewhere. That needs to be tracked. This is where
.csv export and import fail.
4) Rows
One of the simple pleasures of working with a spreadsheet is
the moment when you realize you've forgotten a row, and you just insert
it in the right place in the middle of things. As long as it doesn't
break that absolute reference you put in that other place... Rows are
often inserted or deleted, and tracking that change by individual cell
(a) is tedious, and (b) misses a piece of information. The user was
inserting or deleting a data record of some kind (usually), or adding a
calculation that wasn't there before. This could be noticed as a change.
5) Columns
Of course, anything you can do to rows, you can do to
columns. Again, it may be a whole new record, or more likely, a whole
new attribute across all the records, that you are adding or removing.
New calculations across the entire data set. Storing that change as a
bunch of cell additions turns that useful information into a bunch of
noise.
CXL is aiming to be a smart, spreadsheet-savvy document repository.
Sharepoint and other document management systems can store versioned
spreadsheets and keep the "latest" copy available. CXL will break your
spreadsheet down across at least these five dimensions, tracking each
change where it makes the most sense. The evolution of your document
becomes a story CXL tells back to you. Of course there will be security
and user roles and easy access. But CXL will see into your document,
helping you develop it and protecting you from that change that
necessitates that call to some expensive "expert" who will tell you to
quit using spreadsheets and get a real application already.
I think we can make your spreadsheet into a part of your business that
you don't have to worry about.
dave@northcreek.ca
P.S. The Beta will start in January. We are looking for a few more
businesses who have spreadsheet problems that need fixing. Let me know
if you are one of them!
Categories:
Business
,
Software
Spreadsheets and Databases
When I was a corporate programmer, spreadsheets were evil.
There were too many of them, they contained important and valuable data,
and we (the IT people) couldn't keep them under control.
People would have them on USB sticks, on Dropbox, on their kid's laptop.
And the calculations were stuck in an obfuscated binary format, hard to
get at and hard to reuse.
So why are spreadsheets still around?
1) They are easy. Nothing to a good model or a good project breakdown in
20 minutes. No other tool gives you that much flexibility.
2) They are flexible. Add a column. Add a row. Change the formula on the
third linked worksheet and look at the graph change.
3) They are personal. You can do things on your desktop (or laptop) (or
tablet) that others don't have to see or laugh at you about. You can
spitball a scenario. You can what-if. You can write one version of the
project summary that is not for publication, then actually erase it and
write a not-so-candid version.
So where do spreadsheets live?
Wherever change is happening.
A new company doesn't have the budget or the insight to commit to a
database schema that won't change. It can't know whether this tool or
that tool will match its process. So spreadsheets work during growth.
During disruption, trying to fit the new reality into an ERP package
built for the last decade (and designed during the decade before that)
leads to frustration and error. A spreadsheet tracks the data as things
change around you.
CXL will use a gentle touch. Keep your spreadsheet. Keep your formulas.
We'll find the data in the spreadsheet. We'll track changes as they
happen, whether they are within a cell, a new row, a new column. The
history will be there. And you will be able to manage change.
CXL will do branching. Spreadsheets are personal, and the other person's
template may be a good starting point for them, but not for you. Or
maybe you want to take a model in an entirely different direction.
Branch the document, don't lose the history, and continue on your own
path.
CXL will be easy. Just a file upload, and you have a document in a
secure repository. There will be a public URL if you want to share with
the world. There will be private access tokens that you can hand out.
There will be the option to share only with a certain group of people,
that have to log in using their email address. And for each level, you
can restrict to preview only, download privileges, the right to propose
changes, or document ownership. You can make things complicated if they
need to be. But simple if that's all that you need as well.
Suggestions? Comments? Let me know. Email me at dave@northcreek.ca
to join the Beta in January, or to subscribe to the North Creek mailing
list.
Categories:
Methodology
,
Software
Spreadsheets and the failure of IT
Just finished my 4th day of the Symfony tutorial. It does pack quite a
bit into a one-hour (supposedly) lesson, and if you cut'n'paste the code
from the web page, things move pretty quick. But it looks like a pretty
well-designed project layout, with lots of small folders so that things
are put in the right place, and lots of convention-over-configuration.
I'll keep at this, get through the tutorial, and maybe I'll write that
music management app I've been thinking about once I get a handle on
this.
I actually touched some php today. As a long-time Perl bigot, I wasn't
sure I'd like it, but if you think of it as crippled, purpose-built web
perl, it seems okay. Perl 6 is still not here, so I'll live with what I
have access to on the server (as I said yesterday).
Now for some Magento, then I have to prepare for a real-life
interruption - a ski trip, just when I was generating some momentum with
this coding stuff. I'll bring the MacBookPro, and we'll see how much
bandwidth (and time) I get on the mountains (Fairmont, BC).
Categories:
Software
More Symfony, thoughts on PHP
Welcome back to all of you who have missed me for the past 11 months :-)
I have done a few things - my house is now liveable (in the basement),
and I have a GST # and a few clients. I am now working on a couple of
proposed projects - I need to spend some time up front so that I have a
demo to show them.
1) Magento - a free (to
start with) e-commerce php package. Setting up at http://www.northcreek.ca/store,
this will be a development web store. We'll see how far Magento can go.
2) Symfony - a PHP library
that helps build web apps quickly and cleanly. I want to build a music
management app for Onoway Baptist Church, where I am the worship leader,
and possibly soon the webmaster. Symfony gives me a good starting point.
So why PHP? Well, I have a hosted server with GoDaddy (that you are
reading this on), and they support PHP, not perl, java, ruby, or others.
No root access. So I'll go as far as I can with this setup, then move to
a java-supported server setup when required. In the meantime, I've
installed the Entropy.ch php package from Marc Lyanage (www.entropy.ch)
on my laptop, so that I can develop using php and mysql at home.
I will try to keep things updated - I'm currently on Day 3 of the
Symfony trial. Lots of configuration headaches due to my laptop.
However, with very little description of a model, I now have a database
and a website that edits the database. Hooray for modern web frameworks,
even in PHP! :-)
Categories:
Software
PHP, Symfony, Magento
I spent this morning registering with
Elance,
an online virtual outsourcing broker. I think this is a positive step
for North Creek. Most of the jobs on Elance are bite-sized, and I can
build up my portfolio (from nothing) with a few well-done jobs (and
develop a set of references for the business, not just for David Block,
employee).
In filling out the profile, I discovered
some things about myself. I am not too confident in delivering a
beautiful, dynamic, functional website (using PHP and some CMS), but I
am ready to compete on requirements gathering, architecture, design,
data modelling, and back-end coding. So that is where I can make some
money, and then as I develop a better toolset for graphical front ends
(whether desktop or web), I can start competing in that space as well.
My goals are crystallizing for North
Creek's architecture. I want to write nothing from scratch after a job
has been won - even for the detailed customizations that every client
will expect. I want to build a suite of generic applications that can be
quickly modified to meet the needs of my client base. My goal is 18
hour turnaround - if you tell me in a morning meeting what
you would like to see, I want to be able to provide a working demo (not
a hack) by the next morning meeting. This provides me with a framework
around which to make architectural choices.
In a surprise move, I am quite seriously
considering learning C++. I now have two books (one on generic C++, one
on building with Qt), and have downloaded the relevant frameworks from
Qt that will allow me to write serious applications for multiple
environments. I have spent my entire working life with multi-tier
environments, but that doesn't seem to be a big factor in small
businesses, so this evolution seems to make sense. I will keep this blog
updated as I progress and make more choices.
Categories:
Business
,
Software
Registered with Elance
Well, I don't expect these to be
comprehensive reviews, but just a few notes about the books I have
finished.
I have begun a regimen where I read a
chapter out of a book every day. This keeps a constant velocity, and
lets me get through books that have been on my shelf for years. I have
enough varying interests that I am reading 4 book chapters/day. This
takes some time away from coding right now, but I am learning so much
that I accept the cost. Eventually I will get disciplined enough to do
my reading in the early morning, before the world awakes. I expect that
will be later, though, when the log house is complete and I can stumble
over to my office and sit in a comfortable chair.
The Pragmatic Programmer was written in
1999 by Dave Thomas and Andy Hunt. It has stood the test of time, mostly
because it avoids too much specific technology and focuses on the
principles of effective programming. I can see the influence of this
book in the book I'm working on now, "Agile Software Development:
Principles, Patterns, and Practices" by Robert Martin. "Pragmatic" has
become a code word and has been branded as such in the industry. The
"Pragmatic" authors have taken their own advice and started learning a
new language every year, which is profitable for them, since they often
get to write the book about said new language. They jumped on Ruby
fairly early, if I recall. It makes sense: this book has a lot of Perl
in it, and Perl is so 20th Century.
Metaprogramming is a big deal for them -
getting scripts to write code. This has become standard in the industry,
and .Net and Java annotations do this as part of their internal
structure. However, that, and DRY (don't repeat yourself) are the big
takeaways from this book.
They are fumbling about the agile way, but
XP was just being articulated at the time, so things like Unit Testing
and short iterations were still somewhat controversial. Now they are not
controversial, just not done in practice. Hmmm.
Don't Repeat Yourself is a huge deal. If I
could get my code to that level, I'd be ecstatic. Redundancy is a real
enemy. It is the one thing that killed EJBs, I think. Spring is just so
much more terse. It is also the reason I don't like Hibernate. I have a
database schema that already exists; I don't want to repeat it in
awkward XML. I also don't want to give a single application power over a
database schema - often the data is much longer-lived than the
application. So I want to work from the data out. That's what led me to
the design for DBDB.
Getting good at an editor! What a good
idea! I used to think in Emacs keystrokes, but I've been away from
coding for long enough that the Mac way is more natural now. I don't
know if that will stay, or if I'll get back to Emacs, which has so much
more power available without lifting my fingers off of the keyboard.
With this laptop, I have a touchpad not too far away, which makes things
somewhat easier, but still, keeping the fingers on the home row is
definitely a win overall.
I really appreciate the wisdom in "The
Pragmatic Programmer," but I am nowhere near that level yet. I will keep
working towards it - it feels like I may have to put this one in a
rotation, and read it again in a year or so, to keep the target in front
of me.
In the meantime, I have plenty of other books in front of me.
What did you think of "The Pragmatic
Programmer?" What, you haven't read it yet? Or are you like I was, and
felt so guilty about how you actually work that you never finished it?
Categories:
Books
,
Software
Book Review: The Pragmatic Programmer
I have discovered that I have been away
from coding for a while! My last coding assignment was with EDS(
www.eds.com,
now a unit of HP), working on an existing codebase that dealt with a
mixed Oracle/mainframe backend, with a Java servlet/JSP front end (
http://travis.gov.ab.ca).
This was a familiar environment, since I worked on similar systems at
Novartis (
www.gnf.org) and the
University of Alberta Human Metabolome Database Project (
metabolomics.ca).
However, my work at EDS transitioned into Business Analysis, including a
lengthy stint as a facilitator for a large-scale Smart Card
implementation. While I was gone, the world changed!
The world has experimented with web user
interfaces (UIs), and has found them wanting. In their place, the Rich
Web has gained prominence. This primarily has meant AJAX, and in
practice, the Google Web Toolkit along with other offerings. Adobe has
introduced Adobe AIR, which allows cross-platform desktop development
based on Flash. And the rise of the Mac platform to relevance has
re-invigorated cross-platform library solutions, such as Qt, a unit of
Nokia, which provides Java and C++ development environments that can be
targeted at almost any platform, including Linux, Mac, Windows, and many
smart phones.
So I want to develop custom, easily
modified business software. So do millions (?) of other developers. I
hope that my choice of technology gives me a business advantage. Of
course, my primary business advantage is that I will start by creating a
relationship with my clients, and then will meet their perceived and
actual needs. This means that my choice of platform must allow me to
fulfill those needs within a cost and time structure that allows me to
be profitable.
So, what do I do? Here are some major
factors in my decision-making process (which has not finished yet!):
1) Run on desktop vs. remote server: Do I set up a server and
allow my clients to log in to the server? Or do I send a complete
solution to the client desktop, including a persistence solution and all
business logic?
My experience and my instincts suggest that if I can control the server,
I can improve the experience for the client without the pain of an
upgrade process. This should allow me to iterate rapidly towards a
really good product.
2) Web client vs. Rich client: Should the user interface be
compiled to run on the local machine, or should it be mediated by one of
several possible browsers, over an uncertain network connection? This
seems to be somewhat of a false dichotomy, as long as the client has
decent bandwidth. With my choice of a server backend in #1, the
bandwidth seems to be required. Can a solution such as GWT support
enough functionality to make the browser "disappear?" Or would a Java
client offer enough speed and interactivity to make the browser
interface seem clunky? Perhaps I should let Qt provide the UI framework,
and write once, test everywhere?
3) Client hosting of data: Will the clients be comfortable with
their private business data in the "cloud," or should I sell them on
local appliances that host their solution on their local intranet (which
means in many cases, setting up an intranet)?
The low cost of desktop hardware leads me towards this second option,
where I build and configure a server that hosts the persistent data and
the application, which is served to the client desktop (through
something like Java Web Start, or through GWT, or Adobe AIR). This
allows me to set up backup schemes and encryption to reassure the client
of the security of their data, and gives me a single upgrade point per
client.
Given the explosion of virtual server
environments available, I believe this progression of questions and
answers will lead me to develop a standard appliance that can be easily
loaded into a commodity PC. This appliance will host the applications
required by the client, while the data will be stored on the PC outside
of the appliance, allowing for quick swapping of images when I wish to
push out an upgrade or bugfix, without touching the client data.
I have a lot of experience with MySQL, and
combined with its price point (free!), I plan to stick with it for now
(while watching PostGres et al). Combining MySQL with a JVM and a
servlet container (Tomcat? Jetty? Jboss? I'll decide later... the joy of
standards) gives me a comfortable backend. Now, developing on the front
end for the JVM has gotten interesting, and that's where I'm still
experimenting. GWT? AIR? I will probably pass on AIR for now, as its
advantages are pretty small compared to the other options, and there is
a small upfront cost involved. I'm still happiest with free tools.
I am taking a good long look at Qt, and its
Java implementation, Qt Jambi. That combination gives me a near-native
front end (from C++ compiled into native code) combined with Java (which
I actually have worked in before). If it's not dynamic enough (more on
that in a later post), I will probably invest some time in GWT, and then
fall back to HTML+javascript as a last resort (but the one I actually
know how to do now).
Suggestions?
Categories:
Business
,
Methodology
,
Software
Coding Options
Well, a bit of an update.
I'm coaching a basketball team (my son's Grade 7 team), which takes up
some brain space on Mondays (practice) and Wednesday or Thursday
(games). They are just beginners, but that's fine, so am I :-)
So today I just got started on a database schema, and then had to plan
my offense. The practice went okay, but the kids need way more
repetition before they do the offense naturally. So hopefully they at
least have fun.
My database schema - very basic:
mysql> show tables;
+----------------------+
| Tables_in_northcreek |
+----------------------+
| event |
| participant |
| party |
| time |
| type |
+----------------------+
These tables will be the start of a generic framework that allows me to
create business applications using metadata rather than code. At least
that's the plan. I need a template system for the UI (View), a workflow
system for the Controller, and an extensible, Universal data model for
the Model. My goal is 1 day turnaround from requirements to prototype.
So Monday requirements meeting with a client, Wednesday demo of those
requirements in a production app.
If I can get this to work, then I should be able to undercut commercial
business accounting software and still make money on support contracts.
But we're a ways from having a Quickbooks feature set. That's the
target, though.
I may use Qt as the development framework - cross-platform development
including smart phones and Java. It depends on if I can figure out how
to template a dynamic UI that runs off of metadata. I know I can do that
using web technologies, but that introduces servers and browsers, etc.,
which is not all bad, but might be overkill for a small office.
I've also downloaded Scala to play around with - the alpha geeks have
given it a thumbs up, and it runs in the JVM, so I can integrate it with
everything else out there (including Spring). I'll keep this blog posted
- I just completed a Hello World app that actually ran in Eclipse :-)
Bedtime for the kids. Out.
Categories:
Business
,
Software
A little progress
So I'm attempting to install a commenting system - hope to hear from
some visitors!
We'll see if it works.
I am using blogtools, which hosts the
comments and just requires me to update my templates (well,
Thingamablog's templates) to pop up a comment window.
Doesn't seem to be working yet - I will update soon.
Another attempt? YES!!! I plan to always celebrate little successes
like this one - maybe by breaking for lunch :-)
Categories:
Software
Comments!
I spent most of yesterday looking for a free desktop blogging client
that doesn't rely on a blogging API to post. I have a hosted domain with GoDaddy
(northcreek.ca) and I just want to
ftp a rendered blog entry up to the host, without a lot of complicated
installs (mysql, wordpress,
etc.). I was using Radio Userland,
but it seems to be abandonware (Dave
Winer, I know you're busy, but Radio breaks my Mac!). I rebooted
about 10 times yesterday, when usually I reboot only on Software Updates.
I looked around at Google's suggestions,
and ended up with Thingamablog.
This is a Java application that does exactly what I need. The only
problem was that it was a .jar file with a bunch of associated lib jars,
nothing like a Mac application. Apple has done a good job of making Java
apps look "close enough" to real Mac apps that I can deal with the
strange icons, etc., but I wanted a double-clickable .app bundle that I
could move around.
So I made one! I got a lot of help from a bunch
of websites that
pointed
out the basics,
and I wouldn't want to rely on this app being distributed like this, but
it seems to work so far.
I'll stick with this until (a) I find something better, also for free,
or (b) I write my own. Of course, Thingamablog
is free (GPL), so I can always pick up the source code and make it
better.
I cut'n'pasted an icon from the splash screen - I am not a graphic
artist! - but let's see how Thingamablog does with images:
Splash Screen:
New Thingamablog Icon:
And there we go!
Categories:
Software
Blogging with Thingamablog
Welcome to the North Creek Blog. This will be a record of the creation
of a new software development company in northern Alberta.
My name is David Block. I am starting the company I would like to work
for (Thanks for the idea, Joel).
It will use agile, Test-Driven Development, and it will assemble
Open-Source tools and libraries to create custom applications for small
businesses that have the capabilities of much larger, more expensive
commercial software. I plan to stand on the shoulders of many other
giants, and see what comes of the process.
I have a few ideas of my own. I like working off of a real database
engine, so I will start with a general database schema that will store
most of the nouns in the program. Then, I will build a workflow solution
on top of that schema that thinks in terms of verbs. The verbs will lead
to the dynamic generation of the interface needed.
I have a library of database interaction code that I have developed over
several previous projects, for different employers. The latest name for
it is DBDB. I will use it as the ORM for my application. It is very
lightweight, and requires very little maintenance to adapt to dynamic
database schemas. I don't like maintaining a big xml file - flashbacks
to J2EE.
I will also use a lot of Spring
for setting up the project.
I will try to post reviews of useful books,
projects,
and libraries as I try them out.
Currently, I'm trying to read some of the books that have been sitting
on my shelf for a while: JUnit Recipes, The Pragmatic Programmer, and Uncle
Bob's Agile Software Development. Now I have to try to apply those
principles as a small business owner, working on my own.
Hopefully I'll see you around.
Categories:
Books
,
Business
,
Methodology
,
Software
Introduction