Leonardo ENERGY’s Impact

Assessing the Impact of the Leonardo ENERGY Initiative

Marketing impact

February 18th, 2008 by · 1 Comment · impact

The impact of marketing activities can be subdivided according to the direct relation between the activity and its impact, and the time horizon of the impact.

These variables are obviously correlated. The more remote the impact, the longer its time horizon.

Moving from left to right on the impact scale, complexity of measurement increases. Impact of a sales campaign which results in orders in the short-term should be relatively straightforward to measure. But the impact of marketing activities where an organisation tries to position itself, building reputation through a white paper programme is already more difficult. The 3rd level, which is typically our universe, is where marketing moves beyond the individual organisation, to pursue a grand objective such as regulatory action (or avoiding of) and joint market development.

Measurement in the first column will be done through sales. A good tracking system needs to be in place to determine customer lifetime value.

Measurement in the second column is done through sales trends. Simplistically, one advertises cookies, and observes more sales. Or one advertises an instrument, and observes an increase in leads, closure rate or both.

For the 3rd column, one needs to measure at industry level, observing market trends. The dilemma is that either aggregate statistics are available, but they are at a too high level, and hence noisy since they include many impacts. Targeted statistics for the programme are mostly not available, or not available with the accuracy needed.

A final observation is that the return on marketing investment for direct sales will be modest, but tangible. For remote activities, it’ll be much higher, but largely based on corroborated evidence.

Blogged with Flock

Marketing Measurement Webcast series

February 18th, 2008 by · No Comments · evaluation, impact, tools

Better Management has an excellent series of 6 webcasts so far on Marketing Measurement:
Marketing Measurement Today – BetterManagement.com

When to measure

February 9th, 2008 by · No Comments · evaluation

When you can measure something in numbers, then you know something about it.

Lord Kelvin

Starting a new campaign, it’s almost impossible to make the business case, since you don’t know enough. But to know enough about the market, you need to start a campaign (or do some sterile market research), for which we need a business case …

What ways out do we have?

  • Be lenient on the business case in the early stages
  • Run a pilot project
  • Accept a percentage of budget to be used for ‘exploratory activities’
  • Never start anything new

Tags:

The opportunity to influence

February 8th, 2008 by · No Comments · impact

Standards and regulations (defined broadly) are an increasing part of our programme. These are seldom influenced by single or few organisations. Especially for the complex power system, affecting many users, a wide variety of stakeholders is involved in almost any issue.

This begs the questions:

  • In which issues should we get involved?
  • If we do get involved, how do we assess our impact?

Criteria to get involved could be:

  • the copper potential – often difficult to assess for complex policy issues (e.g. EuP) or even ‘simple’ standards
  • our background knowledge in the subject matter
  • relation to other campaigns we are involved in
  • the opportunity to learn
  • the opportunity to position ourselves

Once we do get involved, our influence will always be indirect. But as with classical education campaigns, one thing is for sure: not providing input will result in no influence. For regulation, the same interim tools then apply: inputs, campaign metrics, reactions, conversions. The latter can be measured by the usual tools: interviews, case studies, surveys.

What do we know of our users?

February 7th, 2008 by · No Comments · branding

  1. About 3000 registered users, with profiles of varying quality. Probably the richest dataset, but largely unstructured. For each user, we also have individual page visits for the past 2 weeks. And we have a user’s subscriptions to LE’s Communities of Practice.
  2. About 11,000 email addresses, with a profile of topics of interest & opening/CTR report of the last 20 email blasts.
  3. The 2006 & 2007 LE web user surveys. About 220 replies in 2006, and 130 so far for 2007.
  4. User feedback as comments. About 1-2 comments per working day. In addition, we have some user posts on the forum.
  5. User questions through the contact form. A few emails per day, on a tremendous variety of topics.
  6. Aggregate statistics per node, download & content stream.
  7. User interaction through webinars.

It should not come as a surprise that above information is largely unstructured, although some structured information is available. Anticipating a suggestion, it is unlikely that integrating these various data will yield anything but a thin result.

So, we know quite a bit. Intelligence on users is dispersed. A few questions we could ask:

  1. What else do we need to know, and how will we collect it?
  2. How can we mine the information we already have into something more useful?
  3. Are we reaching the right users?
  4. Are we catering to their needs?

We need specific answers to above questions. And we should at least have a good luck at the data we already have.

Sense of scoring

February 6th, 2008 by · 2 Comments · evaluation, tools

When does scoring make sense. Suppose you’re an account manager for a geographical region, and have 100 prospects to develop into customers. What you’re likely to do is to define a number of criteria relevant for you to prioritize these accounts. Factors under consideration might be:

  • do you do business with this client in other regions?
  • growth potential
  • size
  • interesting account (business may be low, but the client helps you to push the boundary)
  • …

You’ll do a first ‘quick & dirty’ ranking, that you revisit in the beginning frequently to go through a number of iterations, to come up with a system that sounds like a plan. In the beginning, nonsensical results are likely to come up, which you’ll quickly spot and fine tune. In the self-contained universe of an account manager or small team, this approach makes a lot of sense.

Now suppose that you want to centralise this approach accross account managers, to allocate resources efficiently. Immediately, it gets very political. Every account manager will want criteria in there on which his accounts score well, and you’ll end up with an amalgamate of criteria. Next, gaming will enter into the scoring. If you’re team of account managers scores, you may see ‘reciprocal scoring’. And nonsensical results are not sorted out quickly anymore – they are convenient for anybody who wants to bypass the system, which is only a matter of time.

So lead scoring, campaign scoring, project scoring, which makes a lot of sense for individual portfolio’s, loose their value when used at a too aggregate level.

Blogged with Flock

Relative return on investment

February 5th, 2008 by · 1 Comment · impact

This post is about showing relative ROI.

To show the success of the electrical programme, it may be useful to calculate the cost of reaching each person in the target audience.

We can measure how many people we reach by conducting a survey now and then another (creating two data points) in the future.

To rate the success of the programme, we can then compare it to the cost of using other channels to reach the same number.

Starting the impact white paper

January 30th, 2008 by · 2 Comments · impact

A few quotes relevant at this stage:

“A perfection of means, and confusion of aims, seems to be our main problem.” to measure impact, we need to define LE’s mission. Eventually, it’s copper, but at an intermediate level, it has to be something else.

“Not everything that can be counted counts, and not everything that counts can be counted.” There will always be a belief element in programme impact assessment.

Having been ‘at this early stage’ now for over a year, it’s time to move forward, and make some declarations.

So first, what’s LE about? Recognizing the strong correlation between sustainable energy and copper usage, we want to move the technical fabric of society towards sustainability. Therefore, we promote technical solutions and individual, corporate and public programmes (including regulation) that accelerate this move. The key elements are:

  • promote rather than develop from scratch
  • 2 avenues: technical, engineering solutions and programmatic action

We’ve stated our core business as ‘education’. When targeting the supply chain, we call it professional development. Towards complementors, we call it advocacy.

At a higher abstraction level, we could state our core activity as ‘mind change’. Time for a new quote.

Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.

So what are the levers of mind change (cf Gardner)?

  1. reason – arguing a case. E.g. the discussion webinars.
  2. research – new insights. Hence the importance of knowledge management.
  3. resonance – branding, using the power of language. The statement ‘we need to be more in touch with our users’ will apply forever.
  4. representational redescriptions – metaphors, images and other analogies to change entrenched ideas. E.g. our tutorials, tools, animations, short didactics, …
  5. resources & rewards: at the disposal of governments and corporations, but not for us.
  6. read world events: an external lever, which sometimes can be useful

Clearly our focus is #1 to #4.

On the arena’s of mind change, we are much more restricted. We’re not a government, a formal institute or education establishment. Our main instrument is indirect mind change, through creative works. This can be extremely powerful, cf Picasso’s Guernica or Darwin’s Origin of Species. But it is also the most difficult to appraise. Moreover, still drawing from Gardner, we can strive for ‘Big C’ creativity, but probably have to settle for ‘middle c’.

If above describes our aims, what are our means. We could list campaigns and campaign types, but taking a bit more distance, we have 4 means, each defining its own success metric:

  • communication: broadcasting published work (metrics: views, downloads, links, e-mail subscriptions)
  • engagement: discussion (metric: comments)
  • evangelism: users spreading our words (metric: registered users)
  • partners: collaboration (metric: recruited partners)

Attitudes and behaviour change as a result of programme activity can be measured in a variety of ways:

  • Interviews
  • Surveys
  • Case studies

This leaves copper impact, which can only be measured indirectly, since no end-use trend data exists. We can:

  1. Combine published market data with copper intensity of use data to determine a trend. This can be combined with above programme impact as corroborated evidence.
  2. For engineering solutions, we can perform model-based calculations, and have them evaluated by Delphi panels.

Blogged with Flock

web traffic measurement

January 28th, 2008 by · No Comments · Uncategorized

The debate goes on even more intesely outside of our immediate discussions and is now saying that neither panels nor DMD [directly measured data] are enough – you need both and then some [how about Mass Inference algorithm?].

That last comes from a group/ site called “Quantcast” that seem to offer something close to “..producing refined profiles that accurately represent web audiences…”

http://www.quantcast.com/how-quantcast-works.jsp

That’s their address – do we think they might be worth a try? As far as I have seen so far  it seems to be a free service … but is there anything such as a free lunch?

Tags: ·

re: Our programmes and tonnage impact

January 21st, 2008 by · 2 Comments · impact

Speaking to Hans this morning he reminded me that this input [11.12 of last year] was using the ante-deluvian email format and was never posted where it should have been …ie here. So here it is.

Dear Both:

Apologies for not having inputted sooner.

What I might add to what you have already discussed is as follows:

  • it is essential at the outset to separate out the processes that address measuring attitude and behavioural change and tonnage consequences – the measurement mechanisms are obviously different – equally obviously they need to come together at the end to make any sense of what we are doing.

To address the attitude and behavioural change first. The path I would follow is:

  • For each plan or programme set out where we stand – let’s call this the Rumsfeld approach ie defining what we know and what we think we know [even what we don’t etc]. This statement will be as much something to be proven wrong as it is something to be verified by subsequent intelligence.
  • Define how we want people to change whatever they are doing or thinking as a result of what we do on the basis of the fact that they will be using more copper as a consequence.
  • By sampling to an agreed level of reliability track the changes in attitude and behaviour and do the sums.

Addressing the tonnage consequences of our actions then:

  • I still think that the macro intelligent approach Angelo started is good and could help; perhaps our challenge may be to make it easier for those concerned about how it works to see its value.
  • I agree totally that measuring the micro is out of the question – but what we do know is the volume of copper used for given activities – if we can agree these, again as a benchmark that can be revised up or down as experience/ intelligence grows, then the link between the first process and this is quite clear.

In a way we are talking about a [if not the] logical framework:

  • here’s where we stand
  • if we do x …[programme input]
  • they do y …[attitude / behaviour change]
  • and the consequence is z [Cu tonnage derived as defined above

Tags: