The Wayback Machine - https://web.archive.org/web/20191231040538/https://blog.revolutionanalytics.com/

December 13, 2019

R 3.6.2 is out, and a preview of R 4.0.0

R 3.6.2, the latest update to the R language, is now available for download on Windows, Mac and Linux.

As a minor release, R 3.6.2 makes only small improvements to R, including some new options for dot charts and better handling of missing values when using running medians as a smoother on charts. It also includes several bug fixes and performance improvements.

But big changes are coming to R with version 4.0.0, which is expected to be released not long after R's official 20th birthday on February 29, 2020. (The CelebRation 2020 conference will mark the occasion in Copenhagen.) The R Core team has announced previews of some of the changes, which include:

An enhanced reference counting system. When you delete an object in R, it will usually releases the associated memory back to the operating system. Likewise, if you copy an object with y <- x, R won't allocate new memory for y unless x is later modified. In current versions of R, however, that system breaks down if there are more than 2 references to any block of memory. Starting with R 4.0.0, all references will be counted, and so R should reclaim as much memory as possible, reducing R's overall memory footprint. This will have no impact on how you write R code, but this change make R run faster, especially on systems with limited memory and with slow storage systems.

Normalization of matrix and array types. Conceptually, a matrix is just a 2-dimensional array. But current versions of R handle matrix and 2-D array objects differently in some cases. In R 4.0.0, matrix objects will formally inherit from the array class, eliminating such inconsistencies.

A refreshed color palette for charts. The base graphics palette for current versions of R (shown as R3 below) features saturated colors that vary considerably in brightness (for example, yellow doesn't display as prominently as red). In R 4.0.0, the palette R4 below will be used, with colors of consistent luminance that are easier to distinguish, especially for viewers with color deficiencies. Additional palettes will make it easy to make base graphics charts that match the color scheme of ggplot2 and other graphics systems.

R4 pallette

Many other smaller changes are in the works too. See the NEWS file for the upcoming R release for details.

R developer page: NEWS file for upcoming R release

Posted by at 12:46 in announcements, R | | Comments (2)

November 27, 2019

AI, Machine Learning and Data Science Roundup: November 2019

A roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted recently.

Open Source AI, ML & Data Science News

Python 3.8 is now available. From now on, new versions of Python will be released on a 12-month cycle, in October of each year.

Python takes the #2 spot in Github's annual ranking of programming language popularity, displacing Java and behind JavaScript.

PyTorch 1.3 is now available, with improved performance, deployment to mobile devices, "Captum" model interpretability tools, and Cloud TPU support.

The Gradient documents the growing dominance of PyTorch, particularly in research.

Keras Tuner, hyperparameter optimization for Keras, is now available on PyPI.

ONNX, the open exchange format for deep learning models, is now a Linux Foundation project.

AI Inclusive, a newly-formed worldwide organization to promote diversity in the AI community.

Industry News

Databricks announces the MLflow Model Registry, to share and collaborate on machine learning models with MLflow.

Flyte, Lyft's cloud-native machine learning and data processing platform, has been released as open source.

RStudio introduces Package Manager, a commercial RStudio extension to help organizations manage binary R packages on Linux systems.

Exploratory, a new commercial tool for data science and data exploration, built on R.

GCP releases Explainable AI, a new tool to help humans understand how a machine learning model reaches its conclusions.

Google proposes Model Cards, a standardized way of sharing information about ML models, based on this paper.

GCP AutoML Translation is now generally available, and the GCP Translation API is now available in Basic and Advanced editions.

GCP Cloud AutoML is now integrated with the Kaggle data science competition platform.

Amazon Rekognition adds Custom Labels, allowing users to train the image classification service to recognize new objects with as few as 10 training images per label.

Amazon Sagemaker can now use hundreds of free and paid machine learning models offered in Amazon Marketplace.

The AWS Step Functions Data Science SDK, for building machine learning workflows in Python running on AWS infrastructure, is now available.

Microsoft News

Azure Machine Learning service has released several major updates, including:

Visual Studio Code adds several improvements for Python developers, including support for interacting with and editing Jupyter notebooks.

ONNX Runtime 1.0 is now generally available, for embedded inference of machine learning models in the open ONNX format.

Many new capabilities have been added to Cognitive Services, including:

Bot Framework SDK v4 is now available, and a new Bot Framework Composer has been released on Github for visual editing of conversation flows.

SandDance, Microsoft's interactive visual exploration tool, is now available as open source.

Learning resources

An essay about the root causes of problems with diversity in NLP models: for example, "hers" not being recognized as a pronoun. 

Videos from the Artificial Intelligence and Machine Learning Path, a series of six application-oriented talks presented at Microsoft Ignite.

A guide to getting started with PyTorch, using Google Colab's Free GPU offer.

Public weather and climate datasets, provided by Google.

Applications

The Relightables: capture humans in a custom light stage, drop video into a 3-D scene with realistic lighting.

How Tesla builds and deploys its driving automation models with PyTorch (presentation at PyTorch DevCon).

OpenAI has released the full GPT-2 language generation model.

Spleeter, a pre-trained PyTorch model to separate a music track into vocal and instrument audio files.

Detectron2, a PyTorch reimplementation of Facebook's popular object-detection and image-segmentation library.

Find previous editions of the AI roundup here.

Posted by at 12:37 in AI, airoundups | | Comments (0)

November 12, 2019

AzureR updates: AzureStor, AzureVM, AzureGraph, AzureContainers

Some major updates to AzureR packages this week! As well as last week's AzureRMR update, there are changes to AzureStor, AzureVM, AzureGraph and AzureContainers. All of these are live on CRAN.

AzureStor 3.0.0

There are substantial enhancements to multiple-file transfers (up and down). You can supply a vector of pathnames to storage_upload/download as the source and destination arguments. Alternatively if you specify a wildcard source, there is now the ability to recurse through subdirectories; the directory structure in the source will be reproduced at the destination.

There are revamped methods for getting storage properties and (user-defined) metadata for storage objects.

You can now use Azurite with AzureStor, by creating a service_specific endpoint (file_endpoint, blob_endpoint, adls_endpoint) with the Azurite URL. AzureStor will print a warning, but create the endpoint anyway.

For other changes, see the NEWS.md file.

AzureVM 2.1.0

You can now create VM scalesets with attached data disks. In addition, you can specify the disk type (Standard_LRS, StandardSSD_LRS, or Premium_LRS) for the OS disk and, for a Linux Data Science Virtual Machine, the supplied data disk. This enables using VM sizes that don't support Premium storage.

AzureGraph 1.1.0 and AzureContainers 1.1.2

These packages have been updated to use the new Microsoft Graph operations, introduced last week, for managing app passwords. As a security measure, app passwords can no longer be manually specified; instead they are auto-generated on the server using a cryptographically secure PRNG.

In AzureGraph, the az_app$update_password() method is defunct; in its place are add_password() and remove_password(). Similarly, in AzureContainers the aks$update_service_password() and aks$update_aad_password() methods no longer accept a manual password as an input.

If you use Graph, and in particular if you use AzureContainers to deploy Azure Kubernetes Service clusters, you should update as soon as possible, as the old versions are incompatible with the new Graph API.

If you run into any problems, please open an issue at the GitHub repo in question.

Posted by at 10:30 in Microsoft, packages, R | | Comments (0)

Azure AI and Machine Learning talk series

At last week's Microsoft Ignite conference in Orlando, our team delivered a series of 6 talks about AI and machine learning applications with Azure. The videos from each talk are linked below, and you can watch every talk from the conference online (no registration necessary). Each of our talks also comes with a companion Github repository, where you can find all of the code and scripts behind the demonstrations, so you can deploy and run them yourself.

If you'd like to see these talks live, they will also be presented in 31 cities around the world over the next six months, starting with Paris this week. Check the website for Microsoft Ignite the Tour for event dates and further information.

AIML10 Making sense of your unstructured data with AI

Tailwind Traders has a lot of legacy data that they’d like their developers to leverage in their apps – from various sources, both structured and unstructured, and including images, forms, and several others. In this session, learn how the team used Azure Cognitive Search to make sense of this data in a short amount of time and with amazing success. We discuss tons of AI concepts, like the ingest-enrich-explore pattern, search skillsets, cognitive skills, natural language processing, computer vision, and beyond.

Github repository here. Video here.

AIML10

AIML20 Using pre-built AI to solve business challenges

As a data-driven company, Tailwind Traders understands the importance of using artificial intelligence to improve business processes and delight customers. Before investing in an AI team, their existing developers were able to demonstrate some quick wins using pre-built AI technologies.

In this session, we show how you can use Azure Cognitive Services to extract insights from retail data and go into the neural networks behind computer vision. Learn how it works and how to augment the pre-built AI with your own images for custom image recognition applications.

Github repository here. Video here.

AIML20

AIML21 Developers guide to AI: A data story

In this theater session, we show the data science process and how to apply it. From exploration of datasets to deployment of services - all applied to an interesting data story. We also take you on a very brief tour of the Azure AI platform.

Github repository here. Video here.

AIML21

AIML30: Start building machine learning models faster than you think

Tailwind Traders uses custom machine learning models to fix their inventory issues – without changing their software development life cycle! How? Azure Machine Learning Visual Interface.

In this session, learn the data science process that Tailwind Traders’ uses and get an introduction to Azure Machine Learning Visual Interface. See how to find, import, and prepare data, select a machine learning algorithm, train and test the model, and deploy a complete model to an API. Get the tips, best practices, and resources you and your development team need to continue your machine learning journey, build your first model, and more.

Github repository here. Video here.

AIML30

AIML40 Taking models to the next level with Azure Machine Learning best practices

Tailwind Traders’ data science team uses natural language processing (NLP), and recently discovered how to fine tune and build a baseline models with Automated ML.

In this session, learn what Automated ML is and why it’s so powerful, then dive into how to improve upon baseline models using examples from the NLP best practices repository. We highlight Azure Machine Learning key features and how you can apply them to your organization, including: low priority compute instances, distributed training with auto scale, hyperparameter optimization, collaboration, logging, and deployment.

Github repository here. Video here.

AIML40

AIML50 Machine learning operations: Applying DevOps to data science

Many companies have adopted DevOps practices to improve their software delivery, but these same techniques are rarely applied to machine learning projects. Collaboration between developers and data scientists can be limited and deploying models to production in a consistent, trustworthy way is often a pipe dream.

In this session, learn how Tailwind Traders applied DevOps practices to their machine learning projects using Azure DevOps and Azure Machine Learning Service. We show automated training, scoring, and storage of versioned models, wrap the models in Docker containers, and deploy them to Azure Container Instances or Azure Kubernetes Service. We even collect continuous feedback on model behavior so we know when to retrain.

Github repository here. Video here.

AIML50

Posted by at 09:04 in AI, Microsoft, python, R | | Comments (1)

November 07, 2019

New R Support in Azure Machine Learning

Azure Machine Learning has added support for the R language, it was announced at the Ignite conference in Orlando this week.

A new R package azuremlsdk (available to install from Github now, and from CRAN soon), provides the interface to the Azure Machine Learning service. With R functions, you can provision new computing clusters in Azure, and use those to train models with R and deploy them as prediction endpoints for use from any app. You can also launch R-based notebooks in the new Azure Machine Learning studio web interface, or even launch a complete RStudio server instance on your cloud computing resources. Azure Machine Learning service supports the latest version of R (3.6.1) and all R packages (from CRAN, Github, or elsewhere). The video below from The AI Show demonstrates how it all works:

Azure Machine Learning is also great for teams that have both Python and R expertise. You can even call Python models from R (and vice-versa): in this Ignite 2019 talk (presented by me and Daniel Schneider) we deploy R and Python function as a container services, and call them both from a Shiny app. You can also find the slides and associated code from that talk in this Github repository.

To get started with R in Azure Machine Learning, a good place to start is the tutorial "Train and deploy your first model in R with Azure Machine Learning". If you need an Azure subscription, use this link to sign up for Azure and get $200 in free Azure credits.

Azure Machine Learning: ml.azure.com

Posted by at 04:48 in AI, announcements, Microsoft, python, R | | Comments (2)

November 01, 2019

AzureRMR 2.3.0 now on CRAN

This post is to announce that the latest update to AzureRMR is now available on CRAN. Version 2.3.0 brings several changes to make life easier when managing resources in Azure.

New in this version is a facility for parallelising connections to Azure, using a pool of background processes. Some operations, such as downloading many small files or interacting with a cluster of VMs, can be sped up significantly by carrying them out in parallel rather than sequentially. The code for this is currently duplicated in multiple packages including AzureStor and AzureVM; moving it into AzureRMR removes the duplication and also makes it available to other packages that may benefit. See the vignette for more details.

One side-effect of this change is that loading a current version of AzureStor or AzureVM, along with AzureRMR, will bring up a message in the R console:

> library(AzureRMR)
> library(AzureVM)
Attaching package: 'AzureVM'
The following objects are masked from 'package:AzureRMR':
    delete_pool, init_pool

Similarly, if you load the SAR package, you will receive a warning:

> library(SAR)
Warning messages:
1: replacing previous import 'AzureRMR::init_pool' by 'AzureStor::init_pool' when loading 'SAR'
2: replacing previous import 'AzureRMR::delete_pool' by 'AzureStor::delete_pool' when loading 'SAR'

These messages are because the pool functions in AzureRMR have the same names as those in the other packages. You can safely ignore them; everything will still function correctly, and I'll be submitting updated versions to CRAN in the next few days (as soon as the AzureRMR update propagates to CRAN mirrors).

Other changes in 2.3.0 include:

  • Subscription and resource group objects now have do_operation methods, like resource objects. This allows you to carry out arbitrary operations on a sub or RG, if you know the REST call.
  • AzureGraph is now a direct import, which should help ensure your credentials are consistent for Resource Manager and Microsoft Graph.
  • Error messages should now be much more informative, especially when deploying templates.

If you run into problems, or to send feedback, please open an issue at the GitHub repo.

Posted by at 09:30 in Microsoft, R | | Comments (1)

October 08, 2019

AI, Machine Learning and Data Science Roundup: September/October 2019

A roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted recently.

Open Source AI, ML & Data Science News

Tensorflow 2.0.0 has been released. This major update makes many changes to improve simplicity and ease of use. Tensorflow 1.x models will need to be upgraded for Tensorflow 2.0.

Keras 2.3.0 has been released, the first release of the high-level deep learning framework to support Tensorflow 2.0.

spaCy v2.2 has been released, with retrained natural language models and a new data augmentation system.

greta: an R package to fit complex Bayesian models using Tensorflow as the optimization engine.

Julia 1.2.0 has been released. This minor release of the scientific computing language includes no breaking changes.

Industry News

Facebook and Microsoft launch a research program to detect deepfakes, and Google releases a library of deepfake videos for research.

Facebook open sources Hydra, a framework for elegantly configuring complex Python applications.

Hyperparameter tuning has been added to Facebook's fastText text classifier library.

Google contributes MLIR, the compiler framework for Tensorflow graphs, to the LLVM Foundation.

Google open-sources differential-privacy, a C++ library of algorithms to produce aggregate statistics from sensitive datasets.

Microsoft News

The Vision AI Developer Kit, a Qualcomm IoT camera coupled with an Azure toolkit for AI vision applications, is now generally available.

PyTorch 1.2 is now supported in Azure: Azure ML Service, Azure Notebooks and Data Science Virtual Machine.

Azure Video Indexer adds support for detecting animated characters and multilingual identification and transcription.

Jupyter Notebooks are now generally available in Azure Cosmos DB, supporting all data models including Cassandra, MongoDB, SQL, Gremlin and Spark.

Twelve Cognitive Services are now included in Azure Free,  including Custom Vision, Text Analytics, Translator and Personalizer.

Machine Learning Notebooks for Azure ML Service, including a suite of examples of automated machine learning.

Visual Studio expands testing and debugging support for Python.

Python support in Azure Functions is now generally available.

Learning resources

Python for Beginners, a series of instructional videos and associated code samples from Microsoft.

Natural Language Processing Best Practices & Examples, a collection of Jupyter notebooks and utility functions for text classification, entity recognition and more, from Microsoft.

The CodeSearchNet Corpus, an open database of six million code samples released by Github, with the aim of improving semantic analysis of code and documentation.

Excavating AI, a philosophical essay on the issues related to image selection and labelling in training datasets for computer vision.

Video: How Backpropagation Works, part of Brendan Rohrer's End-to-End Machine Learning School series.

Recorded talks from the useR!2019 conference are available to view on the R Consortium channel.

A paper describes how AI agents may learn "superstitions" from reinforcement learning, recalling Skinner's superstitious pigeons.

Reference implementation for deploying ONNX models with Azure IoT edge, as demonstrated in this episode of the IoT Show.

Applications

Write with Transformer, a handy website from Hugging Face where you can generate text from powerful natural language models.

A new paper describes a method to generate a 3-D "Ken Burns effect" zoom from a single image.

An OpenAI project finds virtual agents learning unexpected strategies in a game of hide-and-seek.

CTRL: the Control Transformer Language model can be used to attribute the source of text, or generate synthetic text on a variety of themes. This new language model from Salesforce claims to be the largest released to date.

Startup Airdoc is using computer vision to detect chronic diseases including diabetes and hypertension from retinal scans of patients in China.

Fashion service Stitch Fix provides a machine-learning based service to recommend items that coordinate with a piece of clothing.

Find previous editions of the AI roundup here.

Posted by at 10:00 in AI, airoundups | | Comments (1)

September 27, 2019

Handling dates and times in R: a free online course

If you ever need to work with data involving dates, times or durations in R, then take a look at this free course on LinkedIn Learning presented by Mark Niemann-Ross: R Programming in Data Science: Dates and Times. Here's the course overview from the introductory video:

When did Mount St Helens last erupt? How many birds migrated south this month? Which month last year was the coldest on average? These are questions related to time. And if you're going to perform calculations on this type of data, then you need date and time data structures. I

I'll show you how to store dates and times, how to retrieve dates and times, and how to add and subtract dates and times to discover the duration of an event or the difference between two dates. I'll show how to handle dates for business research and for financial forecasting. We'll explore functions include with base R for dates and time calculations, and we'll explore packages that provide special ways to work with dates and times. 

The course does assume some basic familiarity with R syntax, but does start from the beginning with the base R date and time classes before moving on to the newer Tidyverse packages for handling date and time data. Each chapter also includes a quiz to test your knowledge.

The course content is free to view, and all the R scripts associated with the course are available to download as well. 

LinkedIn Learning: R Programming in Data Science: Dates and Times

Posted by at 11:07 in advanced tips, courses, R | | Comments (0)

September 17, 2019

Obtaining tokens with AzureAuth inside a Shiny app

by Hong Ooi, senior data scientist, Microsoft Azure

As of version 1.2.0 (released to CRAN late last month), it’s possible to use the AzureAuth package to login interactively to Azure from within a Shiny webapp. Because a Shiny app has separate UI and server components, some changes had to be made to the interactive authentication flows. In particular, the authorization step (logging in to Azure) has to be conducted separately from the token acquisition step.

AzureAuth now provides the build_authorization_uri function to facilitate this separation. You call this function to obtain a URI that you browse to in order to login to Azure. Once you have logged in, Azure will return an authorization code as part of a redirect, which can be captured by the browser. The code is then passed to the get_azure_token function in the auth_code argument, which then completes the task of acquiring the token.

Here is a skeleton Shiny app that demonstrates its use.

Note that this process is only necessary within a web app, and only when using an interactive authentication flow. In a normal R session, or when using the client credentials or resource owner grant flows, you can simply call get_azure_token directly.

Posted by at 00:00 in developer tips, Microsoft, R | | Comments (1)

September 12, 2019

A DevOps Process for Deploying R to Production

I've been at the EARL Conference in London this week, and as always it's been inspiring to see so many examples of R being used in production at companies like Sainsbury's, BMW, Austria Post, PartnerRe, Royal Free Hospital, the BBC, the Financial Times, and many others. My own talk, A DevOps Process for Deploying R to Production, presented one process for automating the process of building and deploying R-based applications using Azure Pipelines and Azure Machine Learning Service. The talk at EARL wasn't recorded, but you can see the slides here, and also watch a slightly shorter version of the talk as it was presented at the useR!2019 conference in Toulouse, below:

If you'd like to try setting up a build process for R yourself with Azure Pipelines, this GitHub repository is a good place to start. It provides a simple example of a model built with R, which gets triggered on check-in to the repository (you can see the builds in Pipelines, here). The README.md file also includes links to useful resources on setting up an end-to-end workflow for machine learning.

GitHub (revodavid):  MLOps with R and Azure Pipelines

Posted by at 08:56 in developer tips, Microsoft, R | | Comments (1)

Next »