The Importance of Email Retention Journaling and Recoverability and why Cloud Solutions Fail

The Importance of Email Retention Journaling and Recoverability and why Cloud Solutions Fail

For most of us, email is simply a means of communicating work.  Its a glorified, bi-directional to-do list with comments. Emails come in, we read, do and delete. Once the work is done, theres usually no need to find the email again. 

Thats all very true except for when something goes wrong and your company gets taken to court. Suddenly then, all those deleted emails are very, very important. 

How Email Discovery could work under Litigation

So, assuming that theres a legal case, such as a lawsuit, that involves your company. You could be asked to produce all the emails within a given period (say, six months) which include certain key phrases -- or perhaps all emails from a now-terminated employee.

In the event that email cannot be produced, you could be fined or worse, you could lose the ability to defend your company in court. 

...and it doesnt stop there, your company might not even be directly involved in the court case but could be dragged in as a third party via a subpoena.

No matter how lawfully YOU run your business, not having adequate email retention is a business risk that you simply cant afford to take.

Why A Restore is Not Enough

From an IT point of view, we dont usually talk about legal things.  Were all about Backups and Restores (and recoverability) as if its simply a matter of getting missing data back.

The theory, for example being that if a user loses their file today and theyve not worked on it for a couple of weeks, then any backup from the last few weeks is sufficient.

In this case, the intention is simply to recover the lost data.

What were finding though is that recoverability is much more than simple data restoration. What if, instead of simply recovering the data, we had to prove that no changes had occurred in it between the recovered date and the date it was deleted.  The only way to do that would be to recover all versions of this (or to have unquestioned data tracking enabled).

Email is very much a dynamic kind of "file".  For example, you might be able to recover a mail from July 7 which was deleted on July 20, via the Backup from July 15, but that doesnt mean that someone didnt reply to that message on July 16 and then delete the reply along with the original message on July 20.

Mail Journaling

Theres only one sure way to demonstrate that youve effectively captured all email;

Have a copy of every single inbound, outbound and internal mail copied to mail storage which does not permit deletion - and retain that mail for the appropriate legal period (not necessarily 7 years) - even if the employee in question has left the company. 

and...

Have auditing facilities in place to protect the mail stores from administrator intervention or unauthorised access and, have a monitoring process watching the store-process to ensure that it doesnt stop.

In our case, weve been using the Veritas solution... but now weve discovered that moving to IBM Verse will prevent us from being able to journal purely internal mail.

Why the Cloud Systems are failing us

In the past, when we had our own mail servers on-site, we could direct outbound SMTP traffic to go via our external archiving partners, our inbound mail could be captured via redirected MX records and our purely internal mail could be captured via Journaling.

With the cloud services, attempting to provide a one-size-fits-all solution, these options are not necessarily available to us. In our case, with IBM Verse, weve been able to sort out inbound and outbound mail mail via the traditional means (after a bit of fiddling) but it turns out that theres no way to journal purely internal mail to an external system (so much for open systems).

We have to abandon our archive solution and go for IBMs offering -- except, of course, that we cant really abandon our old solution because we need to keep it going, possibly indefinitely... unless we migrate it elsewhere (See the Chart at the end of this post).

Ive looked at Microsoft and Google and they seem to have the same problems. Their products dont seem to support external journaling any more (or theyre in the process of depreciating them).

Ive also noticed that since were using cloud services, its no longer possible to restore mail (after the trash has been emptied).  This too is a feature of the three cloud services I looked at.

One thing is certain - If youre looking to put your email in the cloud you MUST subscribe to the cloud mail retention service from the SAME vendor.... and, the choices you make today could be the choices you continue to pay for well after youve migrated to a competitors system. 

Recommended Reading and thinking

The whole Email Retention thing pretty much kicked off in 2002 with the Sarbanes-Oxley Act in the US.  Most of the western world now has an equivalent act in place. If youre not up on that, its good reading.

The whole Hillary Clinton thing is worth reading too - its a bit wider than simply mail preservation but its a good example of the rules around email in action.

There are lots of free whitepapers around on Email Retention. Just do a google search and click on some of the PDFs that come up.

Thinking more widely, we need to be prepared for the next leap in litigation; At some point, the courts are going to start asking people to produce records of instant messaging, posts and comments on collaboration platforms.

Do your staff leaving processes leave their collaborative data intact and allocated to the original owner?  How do you handle "deleted comments"?

How Long do we need to Retain Email?

This excellent chart is from Contural Incs excellent 2007 Whitepaper: How Long Should Email be Saved?  It was sponsored by Symantec who have since moved the business to Veritas.  The chart shows that different types of emails have different retention times.

download file now

Continue reading ...

The Power of Cloud Back up

The Power of Cloud Back up

Cloud back-up, also known as on the internet back-up or online back-up is a strategy for support up information that includes delivering a copy of the information over a exclusive or public system to an off-site hosting server. The hosting server is usually organized by a third-party company, who expenses the back-up customer a fee based on capacity, information or number of customers. In the business, the off-site hosting server might be exclusive, but the chargeback method would be similar.

Online back-up systems are typically built around a customer software program that operates on a schedule determined by the level of assistance the consumer has purchased. If the consumer has shortened for daily backups, for instance, then the program gathers, squeezes, encrypts and exchanges information to the assistance providers web servers every 24 hours. To reduce the quantity of information absorbed and enough it takes to transfer data files, the company might only provide step-by-step back up after the initial full back-up.

Third-party cloud back-up has become popular with small workplaces and home customers because of its comfort. Capital expenses for additional components are not required and backups can be run dark, which means they can be run instantly without guide involvement. In the business, cloud back-up services are mainly being used for preserving non-critical information only. Conventional back-up is a better solution for crucial information that requires a few months to recover. When a lot of information needs to be retrieved, it may need to be delivered on record or some other convenient storage space media.

In short there are two main back-up options: pure back-up and hybrid back-up.

In comparison with hybrid back-up, in which an organization backs up to regionally available storage space, pure back-up is all about delivering duplicates off-site to a cloud provider. With pure back-up, information is duplicated straight to the service providers cloud. With hybrid back-up, on the other hand, information is duplicated to a system with set up storage space that resides between our facilities and the cloud. The hybrid system provides the advantage of retaining information regionally to smooth out transport to the cloud and in case restores are needed.

Pure backup

With pure back-up, agents are set up on machines, with back ups going straight to the service providers cloud. The key benefit of a pure back-up strategy is that it is simple to apply, scalable and very well best for organizations that don�t possess in-house IT department skills. The expenses of these solutions go up or down depending on use and allow organizations the ability to accurately design their back-up expenses in line with projected back-up information set growth. This can eliminate the almost-impossible task of attempting to factor in the full expenses of regional back-up storage space in IT budget forecasts. The key drawbacks with pure back-ups are that it is very much restricted by available information. So, it�s not a strategy best for organizations with huge and complex back-up requirements that would hog online connections.

Hybrid backup

For organizations that produce a fair amount of information and have a requirement for readily available recover functions, cloud back-up suppliers have implemented a multiple strategy. This cloud back-up choice consists of on-site NAS equipment that acts as a regional back-up target and then performs background synchronizations of back-up information places to an exterior cloud. When a recover is required from a recent back-up, the information is sitting ready on-site in the NAS and can be quickly utilized. Alternatively, regenerates from exterior cloud-based back-up places can be recalled from the cloud into multiple NAS on-site storage space on demand and renewed from there.

download file now

Continue reading ...

The Trouble with IBM Connections Cloud

The Trouble with IBM Connections Cloud

Im finding myself being increasingly frustrated by IBM Connections.Cloud. It�s a great product, don�t get me wrong. Arguably the most full-featured of its kind. Unfortunately it�s mired by some terrible executive decisions. 

In fact, the problems are so great that it�s pushing us to investigate other products which are �like� connections but are provided by other vendors. 

I came away from an investigation of Microsoft�s Yammer system yesterday shaking my head and saying, �that�s it. Game over�. 

Yesterday afternoon was one of the few times Ive found myself in the uncomfortable position of recommending Microsoft over IBM. 


Today of course is another day. Im a dedicated IT professional, so when I can�t sleep because a system is overwhelming my thoughts, I get up and make a list of my problems.  So, IBM, it�s 3.30am and this one�s on you � and you owe me at least 2 hours of sleep.

Overall thoughts

I can�t fault IBM�s support teams both locally and internationally. They have been brilliant throughout the whole process. Far better than anything Microsoft or Google have ever managed. For the most part, I can�t even fault the product.  It�s technically sound and it does more or less what it�s supposed to, give or take a few silly limitations. Nope. Most of the blame  goes to the licensing team and to management for some truly horrendous decisions.

On-boarding

The gateway to any system is the on-boarding process. At the very least, you have to get this right. There are two parts to on-boarding;

• On-boarding a Company
• On-boarding Users

IBM Connections.cloud manages to screw both of these processes up, so if nothing else, at least it�s consistent. 

On-boarding a Company

Without covering old ground too much, (even though we still haven�t received a licence for our second company -- after about a month of waiting) there are three things wrong with the company on-boarding process;

Real Companies with ABNs Only Please

IBM feel the need to prove that the company is �real� - for example, in Australia, companies need to have an ABN (Australian Business Number). This is fine for registered companies but what happens if you want to on-board something larger than a company, perhaps an international forum?  A fledgling sub-company?  Good luck to you� if it doesnt have an ABN, it doesnt exist.

Microsoft�s Yammer on the other hand happily on-boards anything that has its own domain. 

A Fast Onboarding Process that Fails to Collect Information

Then there�s the on-line forms process for on-boarding.  It�s really slick and fast but it doesnt actually ask any of the questions that need to be answered. There�s a separate paper and signature based form for that -- and even then, the IBM provisioning team seem to ignore what you write and make their own choices. Again, Yammer wins hands down with a complete process that takes only a few minutes.

Segregated Data Centres

I said before that this was the single failing of IBM Connections.Cloud.  I was wrong, there are many other failings but I�d say this one is still the worst. When your company is created, it gets put (almost randomly) in one of three data centres (the US, Europe or Asia). This would be fine except that communication can�t happen between the data centres.  That�s right, people in the US can�t communicate with people in Asia.  For a platform based on collaboration, blocking messaging is a big, big mistake.

On-boarding a Person

The on-boarding process for a user is much better. It�s a bit wordy and �disclaimery� and a bit disjointed too but I would have said it was pretty good - if I hadnt seen the Yammer on-boarding process.  The two big flaws with personal on-boarding are;

• Single-Emails: If youve already used your email address somewhere else in connections, for example at one of the other data centres, then it�s �dead�. It�s tied to that data centre and as mentioned previously, there�s no communication between data centres. This might be fine if you have a single organisation collaborating together but if you�re an association which collects members and guests from other organisations around the world, it makes business impossible. 

• A lack of Wizardry: The Yammer on-boarding process used some very slick screens to navigate the new user through setting up their account, joining companies, adding friends, setting up their profile and introducing the system.  The IBM on-boarding process simply dumps the user in connections in the hope that they�ll figure out where to go to from there. 

Friends and @Mentions

This was bit that �broke� the system for our people.  They presumed that that, as with most social platforms, they could add guests and then draw their attention via @Mentions.  This works extremely well within companies, so clearly there�s no technical problem with the @Mention system itself.

Unfortunately, without some very, very painful user-based processes, you can�t @Mention anyone who is not a member of your company.  You can�t @Mention guests even though they are members of your community.

The Worst Friend-Request System Ever

In order to @Mention a guest, you need to �Friend Request� them first. Then they have to accept. This might not be so bad but the Friend Request system is absolutely terrible. In order to Friend Request someone, you need to go into your personal contacts and add the person manually to your contacts. To do this, you need to type in at least their name and email address. If you don�t know which email address they�re using for connections, then you can�t add them.

Once people are added to your contacts list, you then need to go in and individually send them �invitations to connect�, and then you have to wait for them to accept.  By this time, your requirement to @mention them has pretty much expired.

Of course, I found a way to upload a CSV file which takes a little of the pain out of the first part of the process but considering that Microsoft�s Yammer lets you talk directly to members in your community without any further action, I think it�s too little, too late.

Broken by Design

The worst part about this however is that it was apparently �broken by design�.  The earlier versions of IBM connections.cloud were far more open and for legal or other reasons, those loopholes were closed. This means that there is no technical reason why the system couldnt be better.  It also means that because the �holes� were clumsily closed, the system tends to go to horrible error-message screens when you hit parts of the old functionality.

This is simply a very bad choice of reaction from IBM. A much better choice would be to change the person profiles so that you could have a choice of;

• A community-public edge to your profile (ie visible to all - @mentions within shared communities without needing explicit friendships)
  
• A completely hidden to all profile (ie: the current friend request system)
  
• A list of Trusted domains or companies (ie: @mentions restricted to domains/companies and profiles visible to certain domains/companies only).

All existing Connections.cloud profiles should default to �completely hidden� and all members of the community should be sent a message explaining the differences and how to change it. This would fix the problems in the current system.  From there, all new users could be asked to make the choice as part of the user on-boarding process. The default setting could be set at the company level too.

It�s a simple fix that IBM needs to implement ASAP.

The Capabilities of Guests

One of the most important features of IBM Connections is the Guest System.  The idea that you can invite a guest to connections in order to exchange files with you or participate in your communities.

Apart from the guest problems Ive discussed earlier;

• A sloppy on-boarding process 
• Inability to invite guests if their email has been used at another data centre
• Inability to @mention guests without completing a process too complex for new users

There�s also a few other issues;

Lack of Status Updates

One of the first things I noticed about Yammer was that guests have a status update bar. That�s right, guests can post updates (which can include @mentions and hashtags).  In fact, Yammer encourages guests to introduce themselves when they are added to a community. Connections on the other hand does not allow guests to post status updates.

No Collaborative Document Editing for Guests

Then there�s collaborative document editing. If you�re a fully paid subscriber of IBM Connections, you can edit documents using the built in �IBM Docs� editor.  It�s not a particularly good editor, not compared to Office 365 online or the even better (in my opinion) Google Docs but it does the trick.

Unfortunately, if you�re a guest, you are denied the use of these tools and have to download documents, edit them and then re-upload them.

Initially I was okay with this but then I looked a Yammer. Yammer lets people use the online versions of Office 365

Really?  If Microsoft can run their flagship cash-cow product on a freemium model for Yammer users then IBM must be able to offer IBM Docs to guests. In a world full of free apps (thank you Google Docs) and quality open source alternatives (thank you Libre Office), who pays for the bare essentials of content creation these days?

This is simply unacceptable. 

The Outline Screen and Clutter

The last major flaw of IBM Connections.cloud is, in my opinion, the outline screen. As I said before, connections is far more fully-featured than any of the competing systems. It has great sub-systems like wiki�s, blogs, surveys and activities � and it seems that new modules are coming all the time.

There�s nothing really wrong with these options. It�s all a matter of the outline screen.

The focal point of any community is its start page and in IBM connections, you can choose from several options for the start page. The Wiki is offered as an option but it�s restricted to the welcome page -- and you can�t have any �side apps�, such as tags and calendars.

The outline page is really the best start page of the lot. Unfortunately, there are a few major flaws;

Graphics cant be Directly Added to Outlines

Despite years of badgering (Ive read the forums), you can�t add graphics to the outline, at least not directly. You have to upload them to the Wiki (or files if you enjoy lots of broadcasts) and then link to them from there. Since the rest of connections seems to support uploads, it makes no sense why the one part that would benefit from this feature misses out.

All Components (except updates) Must be Shown on the Outline

If you decide to have a component in your community, for example a survey or a wiki, then you MUST show it on the outline screen -- even if it�s not relevant at the time. You can collapse these items to start with but they still expand for new users.  There�s an option to �hide� these items but hiding them essentially removes them from your community. If you�re going to use the outline screen, you�re going to have to get used to clutter.

Compare this to the Yammer screens, they�re calm, clean and easily navigable. 

Community Status Updates Cannot be shown on the Outline

You can�t include community status updates on the outline screen. This is pretty amazing, as one of the most important things about any community is the status updates.  You�d think that the outline layout which introduces far too much clutter would at least be all-inclusive but it�s not. Status updates are simply not available there. It�s a shame because people have been asking for them too, for years, in the IBM forums.

No HTML in the Outline

If theres one part of Community that would benefit from HTML, its the outline page. This would allow you to integrate your own systems (via iFrames and other devices) and it would allow you to use widgets from other sources. More importantly, having HTML would allow you a lot more control over the layout of the outline.

Where to Now?

I really can�t be sure where we are going now.  For a while there I was so sure that our future was going to be IBM Connections but it�s really not looking like the product will meet our needs. It�s a big decision to make because it won�t just affect connections going forward. It will affect all of our systems, our domino systems, the intended move to bluemix and our xpages development plans.

If nothing else, I�m at least reassured. If we lose IBM now, it won�t be for lack of trying and it won�t be because people made an emotional decision to go to Microsoft.  It will simply be that IBM has �mismanaged� themselves into delivering a sub-par set of rules around an otherwise quality product.

download file now

Continue reading ...

The Future of Cloud

The Future of Cloud

Bold predictions on the future cloud - notes from The Future X Network: A Bell Labs Perspective

Big picture

The cloud era started with salesforce.com (1999) and AWS (2006).

The first cloud was built on a regionally centralized cloud, with data centers in key regions (North and South America, Europe, Middle East & Africa, Asia Pacific)

Centralization of cloud enables economies of scale but is ill-suited to support latency and bandwidth requirements of certain applications.

We need a new "global-local cloud" that will provide �local delivery� while maintaining global reach.

This trend is already underway with content delivery networks (CDN).

The new global-local cloud  will allow optimal placement of applications by considering the needs of the services. The global-local cloud will be a self-learning, automated and self-optimizing digital fabric for the new era.

Latency

Latency is a major issue for content delivery. This is a direct result of the limitations imposed by TCP. 

Maximum throughput (bits/second) = TCP Window Size (bits)/ round-trip latency (seconds).

With a typical TCP window size of 64 kbytes, and round trip time of 100 ms, the TCP throughput is clamped at 5 Mb/s. The throughput requirements for a 1080p stream and 4K are 6 Mb/s and 15 Mb/s respectively.

If latency is reduced to 20ms, the throughput  can be improved to 25Mb/s. CDN networks have responded to these facts by providing distributed architecture.

Akamai, a CDN network, operates 170,000 caches in 1300 public and private networks in 102 countries.

Large content providers have started to establish commercial agreements directly with local and regional network operators. Bell Labs consulting predicts that an increasing number of global providers will leverage the proximity of the local service providers. It is estimated that by 2025, 60% of cloud servers will be located in edge locations.

Dynamic Scalability

The virality of the social media has led to dynamic popularity of certain content. The "Budweiser Puppy Love" commercial spread on social media and received 13 million views on just the first day. 

Cloud services must be able to handle this highly dynamic traffic profile, which requires advanced monitoring, automation and orchestration. SDN solutions will enable creation of fully dynamic data centers. The tools and technologies needed to build this new cloud are still beginning to emerge.

Future

By 2020, the cloud will transform into a true hybrid of global and local resources. The localization of the cloud will be driven by two factors. First, the explosion of the unicast streaming video. Second, emergence of hyper-local, high-performance and low-latency enterprise applications.

According to Nielsen, traditional TV viewing is gradually declining while unicast streaming services like Netflix is increasing. The network architecture needs to shift focus from broadcast delivery of content to on-demand unicast delivery.

There is a also a new need of doing the local processing of high-bandwidth streams for video analytics. Real-time analytics requires sending many unicast streams across the network but in the opposite direction from the streaming video delivery for entertainment. Localization of the cloud resources at the edge of the network allows processing to be located near clusters of devices, reducing network traffic while maintaining the benefits of centralized analytics.

The global-local cloud can decrease the latency to single digit millisecond range enabling new class of applications that require tight feedback control loops. One such application is permitting humans to interact with virtual objects rendered in the cloud as if they are physically co-located. Today such capabilities are only possible through the use of significant intelligence in the endpoint devices.

Both virtual machines and containers will address different virtualization needs and will continue to play an important role in the local edge clouds and data centers.

Intelligent automation and orchestration based on the past and present traffic patterns would be an important factor in the new clouds.

The networking in the cloud will be enabled by software defined networking (SDN). Typically, a network service is comprised of multiple functions known as a service chain. In the global-local cloud, functions in a service chain may be distributed across the cloud. 

The new business model

The global-local cloud will provide low-latency services with global accessibility to users regardless of their location. There will be global service providers (GSPs) and local service providers (LSPs). The relationship between GSPs and LSPs may follow one of the four following models:

• LSPs provide infrastructure for GSPs
• LSPs provide hosted as-a-service function to global cloud services
• GSPs provide local infrastructure
• LSPs provide global services

By 2020, it is likely that we will see a mix of all of these scenarios. However, GSPs will be of limited number (say 10) because of the cost involved in building a global infrastructure.

By 2020, majority of services must globalize their reach but localize the delivery to be acceptable for a highly mobile consumer base. For high-performance and mission-critical services, local delivery will become essential. The end result of this revolution will be a creation of an eco-system in which a handful of major global-local service providers leverage the resources and services of 100s of local-regional service providers.

download file now

Continue reading ...