Calling Esri Out On Claims Of Being Good “Open Standards” Citizens

Source and Credit: Cameron Shorter, Geospatial Solutions Manager, LISAsoft

21st April 2015 

I’m calling Esri out on their claim to be good “Open Standards” citizens. Esri are again abusing their market position to compromise established Open Spatial Standards, as described in an Open Letter from the OSGeo community.

It starts:

We, the undersigned, are concerned that the current interoperability between LiDAR applications, through use of the open “LAS” format, is being threatened by Esri’s introduction and promotion of an alternative “Optimized LAS” proprietary format. This is of grave concern given that fragmentation of the LAS format will reduce interoperability between applications and organisations, and introduce vendor lock-in.

To be clear, Esri has extended LAS to create “Optimized LAS” which provides near identical features and performance to the existing and open LASzip format, both of which provide faster access and smaller file sizes than the LAS format. However, rather than collaborate with the open community, as has been repeatedly offered, “Optimiszed LAS” has been developed internally to Esri. It is neither published, nor open, which provides both technical as well as legal barriers for other applications reading and/or writing to this proprietary format. This creates a vendor lock-in scenario which is contrary to the principles of the Open Geospatial Consortium, the OSGeo Foundation, and many government IT procurement policies.

Esri responded to the open request to avoid fragmenting LiDAR standards with the following motherhood statement, which doesn’t actually answer the key questions:

Regarding Dr. Anand’s concerns and the referenced letter below:

Esri has long understood the importance of interoperability between systems and users of geographic information and services. Esri has participated in the development of national, information community, OGC, and ISO TC 211 standards from the development of the US Spatial Data Transfer Standard in the 1980s through the development of OGC Geopackage today.

As a sustaining member of ASPRS and a Principle member of OGC, Esri would gladly participate in efforts to further the development of open LIDAR and point cloud standards. Keep in mind that ASPRS owns and maintains LAS, along with other spatial information standards, and would have the lead in moving it into  OGC or ISO TC211 for further work if they so desired.

Esri will continue to support and use the ASPRS LAS standard; the Optimized LAS (see FAQ at https://github.com/Esri/esri-zlas-io-library) is not intended to replace LAS but to enhance access to remotely stored LIDAR information for our users.

Lets refute Esri’s statement line by line:

Esri has long understood the importance of interoperability between systems and users of geographic information and services.

Nice motherhood statement. Notice that Esri carefully selects the words “understood the importance” rather than “we commit to implementing”.

Esri has participated in the development of national, information community, OGC, and ISO TC 211 standards from the development of the US Spatial Data Transfer Standard in the 1980s through the development of OGC Geopackage today.

More motherhood words talking about standards in general without answering the specific question about fragmenting of the LAS standard.Esri also failed to mention Esri’s embarrassing attempt to get the GeoServices REST API approved as an OGC standard, despite it competing directly against existing OGC standards. (Esri would have benefited by this, at the expense of geospatial users and competing products.)

The OGC community backlash led to a retraction of GeoServices REST API from the OGC, and a reformation of OGC processes to avoid a similar situation happening again.

As a sustaining member of ASPRS and a Principle member of OGC, Esri would gladly participate in efforts to further the development of open LIDAR and point cloud standards.

Nice statement, without any quantifiable commitment. Will Esri put it into practice? Track record suggests otherwise. As explained by Marin Isenburg, Esri has talked a lot about collaboration and being open, while in parallel creating a competing proprietary format. If Esri were seriously committed to open LiDAR standards, Esri would publish “Optimized LAS” under an Open License, and/or take “Optimized LAS” through a standards development process such as provided by the OGC. Esri would have also build upon the prior LASzip format rather than redeveloping equivalent functionality.

Keep in mind that ASPRS owns and maintains LAS, along with other spatial information standards, and would have the lead in moving it into  OGC or ISO TC211 for further work if they so desired.

Again, if Esri had the best interests of ASPRS and Open Standards in mind (as you would expect from a sustaining member), then we’d expect Esri to donate their LAS improvements back to the ASPRS for safe keeping. Why is Esri keeping such improvements in an Esri proprietary format instead?

Esri would be also lobbying ASPRS to accept improvements to the LAS format. Has this happened? Lack of public discussion on this topic suggests otherwise?

Esri will continue to support and use the ASPRS LAS standard the Optimized LAS (see FAQ at https://github.com/Esri/esri-zlas-io-library) is not intended to replace LAS but to enhance access to remotely stored LIDAR information for our users.

Esri is sidestepping the issue. The LAS standard needs improvements. These improvements have been implemented by the open LASzip format and also by Esri’s proprietary Optimized LAS. One should be incorporated into a future LAS standard.

The question Esri fails to answer is why does Esri refuse to work in collaboration with the open community? Why hasn’t Esri published their own Optimized LAS format instead of improving an existing standard format?

Esri’s FAQ, explains that esri-zlas-io-library is stored on github under the Apache license, which would make you think the code is Open Source and hence the Optimized LAS format could be reverse engineered. This is not the case. Esri has only licensed the binaries under the Apache license such that it can’t be reverse engineered or improved by the community. By the OSI definition, this is not Open Source Software.

So I’m calling Esri out on their claim to being supporters of Open Standards. Please Esri, either clean up the way you behave, or come clean and admit that Esri abuses its market position to undermine Open Standards.

Using Postgres to Integrate MongoDB, Hadoop and Others with FDWs

Presenter: Bruce Momjian, Co-Founder of the PostgreSQL Global Development Group and a Senior Database Architect at EnterpriseDB

Date: Wednesday, April 29 at 8:00am EDT / 2:00pm CEST

A powerful feature in Postgres called Foreign Data Wrappers lets end users integrate data from MongoDB, Hadoop and other solutions with their Postgres database and leverage it as single, seamless database using SQL.

Use of these features has skyrocketed since EDB released to the open source community new FDWs for MongoDB, Hadoop and MySQL that support both read and write capabilities. Now greatly enhanced, FDWs enable integrating data across disparate deployments to support new workloads, expanded development goals and harvesting greater value from data.

About The Presenter: Bruce Momjian is the Co-Founder of the PostgreSQL Global Development Group and a Senior Database Architect at EnterpriseDB.

Target Audience: This presentation is intended for IT Professionals seeking to do more with Postgres in his/her every day projects and build new applications.

Register For The Webinar Here

Corporate Open Source Participation Reaches All TIme High

Source and Credit: April 16, 2015 | Posted in: Future of Open Source | by Bill Weinberg

Over the last decade, companies have increasingly leveraged open source software (OSS) solutions and participated in OSS communities to reduce costs, speed development, and drive innovation.

The growth of projects has made open source more accessible to enterprises and OEMs, and multiplied its impact across many industries. The results from the 2015 Future of Open Source Survey – sponsored by Black Duck Software and North Bridge – confirm this trend, that the rate of adoption and participation by companies of all sizes has reached an all-time high.

Even end users and companies that have historically bucked this trend, lagging behind with legacy, proprietary technologies, are participating in open source projects. This new embrace of open source arises from the realization that organizations face a competitive disadvantage if not involved in open source development.

While the 2015 Future of Open Source Survey found that OSS has become the default approach for the majority of organizations, it also exposed the scary fact that most have no formal management for open source use or policies and procedures to monitor potential open source-related security and compliance risks.

Record Corporate Use of Open Source

Corporate use of open source has been on the rise for years, and this year’s Future of Open Source Survey results serve as further proof of that trend. Seventy-eight percent of this year’s respondents said their companies run part or all of operations on OSS, with 66 percent reporting their company creates software for customers built on open source – up from the 42 percent in 2010 who said they used open source in the running of their business or IT environments. A staggering 93 percent stated their organization’s usage of open source had increased or remained the same in the past year.

Full Story Here…..

Postgres Outperforms MongoDB and Ushers in New Developer Reality

Originally Published: September 24th, 2014

Story Source and Credit: Marc Linster, Senior Vice President – Products and Services, EnterpriseDB

The newest round of performance comparisons of PostgreSQL and MongoDB produced a near repeat of the results from the first tests that proved PostgreSQL can outperform MongoDB. The advances Postgres has made with JSON and JSONB have transformed Postgres’ ability to support a document database.

Creating document database capabilities in a relational database that can outperform the leading NoSQL-only solution is an impressive achievement. But perhaps more important is what this means to the end user – new levels of speed, efficiency and flexibility for developers with the protections of ACID compliance enterprises require for mission critical applications.

Postgres vs. Mongo

EnterpriseDB (EDB) began running comparative evaluations to help users correctly assess Postgres’ NoSQL capabilities. The initial set of tests compared MongoDB v2.6 to Postgres v9.4 beta, on single machine instances. Both systems were installed on Amazon Web Services M3.2XLARGE instances with 32GB of memory.

Full Story and Graphs Here….

Wall Street And The Mismanagement Of Software

Orginally published August 08, 2012

Story Source and Credit: Robert Dewar, Dr. Dobb’s Bloggers

Last week, an error in some automated high-frequency trading software from Knight Capital Group caused the program to go seriously amok, and when the cyberdust cleared, the company was left barely alive, holding the bill for almost a half-billion dollars to cover the erroneous trades. Much of the ensuing uproar has cited the incident as rationale for additional regulation and/or putting humans more directly in the decision loop.

However, that argument is implicitly based on the assumption that software, or at least automated trading software, is intrinsically unreliable and cannot be trusted. Such an assumption is faulty. Reliable software is indeed possible, and people’s lives and well-being depend on it every day. But it requires an appropriate combination of technology, process, and culture.

In this specific case, the Knight software was an update that was intended to accommodate a new NYSE system, the Retail Liquidity Program that went live on August 1. Other trading companies’ systems were able to cope with the new NYSE program; Knight was not so fortunate, and, in what was perhaps the most astounding part of the whole episode, it took the company 30 minutes before they shut down the program. By then, the expensive damage had been done.

It’s clear that Knight’s software was deployed without adequate verification. With a deadline that could not be extended, Knight had to choose between two alternatives: delaying their new system until they had a high degree of confidence in its reliability (possibly resulting in a loss of business to competitors in the interim), or deploying an incompletely verified system and hoping that any bugs would be minor. They did not choose wisely.

With a disaster of this magnitude—Knight’s stock has nosedived since the incident—there is of course a lot of post mortem analysis: what went wrong, and how can it be prevented in the future.

The first question can only be answered in detail by the Knight software developers themselves, but several general observations may be made. First, the company’s verification processes were clearly insufficient. This is sometimes phrased as “not enough testing” but there is more to verification than testing; for example source code analysis by humans or by automated tools to detect potential errors and vulnerabilities.

Second, the process known as hazard analysis or safety analysis in other domains was not followed. Such an analysis involves planning for “what if…” scenarios: if the software fails—whether from bad code or bad data—, what is the worst that can happen?

Answering such questions could have resulted in code to perform limit checks or carry out “fail soft” procedures. This would at least have shut down the program with minimal damage, rather than letting it rumble on like a software version of the sorcerer’s apprentice.

The question of how to prevent such incidents in the future is more interesting. Some commentators have claimed that the underlying application in high-frequency trading (calculating trades within microseconds to take advantage of fraction-of-a-cent price differentials) is simply a bad idea that frightens investors and should be banned or heavily regulated.

There are arguments on both sides of that issue, and we will leave that discussion to others. However, if such trading is permitted, then how are its risks to be mitigated?

To put things in perspective, in spite of the attention that the incident has caused, the overall system—the trading infrastructure—basically worked. Certainly Knight itself was affected, but the problem was localized: we didn’t have another “flash crash.” We don’t know yet whether this is because we got lucky or because the “circuit breakers” in the NYSE system were sufficient, but it’s clear that such an error has the potential to cause much larger problems.

What is needed is a change in the way that such critical software is developed and deployed. Safety-critical domains such as commercial avionics, where software failure could directly cause or contribute to the loss of human life, have known about this for decades. These industries have produced standards for software certification that heavily emphasize appropriate “life cycle” processes for software development, verification, and quality assurance.

A “safety culture” has infused the entire industry, with hazard/safety analysis a key part of the overall process. Until the software has been certified as compliant with the standard, the plane does not fly. The result is an impressive record in practice: no human fatality on a commercial aircraft has been attributed to a software error.

High-frequency automated trading is not avionics flight control, but the aviation industry has demonstrated that safe, reliable real-time software is possible, practical, and necessary. It requires appropriate development technology and processes as well as a culture that thinks in terms of safety (or reliability) first. That is the real lesson to be learned from last week’s incident. It doesn’t come for free, but it certainly costs less than $440M.

Robert Dewar is the president and CEO of AdaCore. He is the principal author GNAT, the free software Ada compiler, and earlier of the Realia COBOL compiler.

MYGEOSS First Call For Innovative Apps In The Environmental And Social Domains

Story Source and Credit: European Commission Digital Earth Lab – MYGEOSS

Context and Challenge

MYGEOSS is launching an open call for development of innovative applications (mobile or web-based) using openly available or crowd-generated data in different domains addressing citizens’ needs. The pool of open data for use includes but is not limited to the Data Collection of Open Resources for Everyone ( GEOSS Data-CORE ) made available by the Group on Earth observation (GEO) through the Global Earth Observation System of Systems (GEOSS), as well as open data from EU-funded projects.

The Focus

The focus of this call will be on developing applications that will provide users with quantitative or qualitative information on the changing environment, e.g. change detection in climate, biodiversity, water bodies, coastal areas, built environment, green areas, forestry, agricultural land and crops, and atmospheric composition. Other areas of application will be considered providing they address broad environmental or social themes across geographic scales.

The Evaluation Criteria

The entries will be evaluated by an international panel co-chaired by the European Commission Joint Reasearch Centre (JRC) and Directorate General Research and Innovation (RTD) and including representatives of the public and private sectors and academia.

Criteria for evaluation for the application will include ease of use of the apps for non-expert users, innovative characteristics of the proposed application, contribution to environmental or social objectives including active citizen participation in data collection and analysis, portability on different platforms, scalability, and ease of translation in different languages. Applications relying on dynamic databases with relevant update, and demonstrating sustainability over time will be particularly encouraged.

The Awards

The 10 best applicants will be awarded contracts by the JRC for a maximum of € 14.000 to develop the applications further, and take them to the stage of first public release within 3 months of signing the contracts.

The winners will also be invited to present their applications to the GEO Plenary meeting in Mexico City during the week of the 9-13 November 2015. This will give visibility to the winning teams as the GEO Plenary and Ministerial Conference will be attended by senior representatives of 97 countries, 87 international organisations and the private sector. Travel and daily allowance will be supported by the European Commission for 1 person from each winning team.

Full Details Here….

Goodbye MongoDB, Hello PostgreSQL

Story Credit and Source: March 10, 2015, by Yorick Peterse

Olery was founded almost 5 years ago. What started out as a single product (Olery Reputation) developed by a Ruby development agency grew into a set of different products and many different applications as the years passed. Today we have not only Reputation as a product but also Olery Feedback, the Hotel Review Data API, widgets that can be embedded on a website and more products/services in the near future.

We’ve also grown considerably when it comes to the amount of applications. Today we deploy over 25 different applications (all Ruby), some of these are web applications (Rails or Sinatra) but most are background processing applications.

While we can be extremely proud of what we have achieved so far there was always something lurking in the dark: our primary database. From the start of Olery we’ve had a database setup that involved MySQL for crucial data (users, contracts, etc) and MongoDB for storing reviews and similar data (essentially the data we can easily retrieve in case of data loss). While this setup served us well initially we began experiencing various problems as we grew, in particular with MongoDB. Some of these problems were due to the way applications interacted with the database, some were due to the database itself.

For example, at some point in time we had to remove about a million documents from MongoDB and then re-insert them later on. The result of this process was that the database went in a near total lockdown for several hours, resulting in degraded performance. It wasn’t until we performed a database repair (using MongoDB’s repairDatabase command). This repair itself also took hours to complete due to the size of the database.

In another instance we noticed degraded performance of our applications and managed to trace it to our MongoDB cluster. However, upon further inspection we were unable to find the actual cause of the problem. No matter what metrics we installed, tools we used or commands we ran we couldn’t find the cause. It wasn’t until we replaced the primaries of the cluster that performance returned back to normal.

These are just two examples, we’ve had numerous cases like this over time. The core problem here wasn’t just that our database was acting up, but also that whenever we’d look into it there was absolutely no indication as to what was causing the problem.

The Problem Of Schemaless

Another core problem we’ve faced is one of the fundamental features of MongoDB (or any other schemaless storage engine): the lack of a schema. The lack of a schema may sound interesting, and in some cases it can certainly have its benefits. However, for many the usage of a schemaless storage engine leads to the problem of implicit schemas. These schemas aren’t defined by your storage engine but instead are defined based on application behaviour and expectations.

For example, you might have a pages collection where your application expects a title field with a type of string. Here the schema is very much present, although not explicitly defined. This is problematic if the data’s structure changes over time, especially if old data is not migrated to the new structure (something that is quite problematic in schemaless storage engines). For example, say you have the following Ruby code:

Full Story Here…..