1. Driving PDI Project Success with DevOps
For versions 7.x, 8.x, 9.0 / published March 2020

This document introduces the Pentaho Data Integration DevOps series: Best Practices documents whose main objective is to provide guidance on creating an automated environment where iteratively building, testing, and releasing a Pentaho Data Integration (PDI) solution can be faster and more reliable, resulting in a high-quality solution that meets customer expectations at a functional and operational level.

Audience: Pentaho Administrators , developers, and architects, as well as IT professionals who help plan software development. 

Starting your Data Integration (DI) project means planning beyond the data transformation and mapping rules to fulfill your project’s functional requirements. A successful DI project proactively incorporates design elements for a DI solution that not only integrates and transforms your data in the correct way but does so in a controlled manner.

Audience: Pentaho developers or anyone who is interested in setting up and improving PDI projects.

3. Continuous Integration with Pentaho Data Integration 
For versions 7.x, 8.x, 9.0 / published March 2020

This document introduces the foundations of Continuous Integration (CI) for your Pentaho Data Integration (PDI) project. It is the third document in the PDI DevOps series, and provides example and instructions geared toward a situation where you are using Git as a code repository, Jenkins as an automation server, and Junit as the test framework. You can use this as a model to build your own configuration using the same principles found throughout the series.

Audience: Pentaho Administrators , developers, and architects, as well as IT professionals who help plan software development. 

PDI Design Guidelines will show you how to create transformations and jobs for maximum speed, reuse, portability, maintainability, debugging, and knowledge transfer.

Audience: Pentaho ETL developers and architects, or anyone who is interested in learning PDI development patterns.

PDI contains step naming standards and help with choosing the appropriate steps in certain situations.

Audience: PDI users or anyone with a background in ETL development who is interested in learning PDI development patterns.

Lookups, joins, and subroutines for PDI are covered in this document through best practices.

Audience: PDI users or anyone with a background in ETL development who is interested in learning PDI development patterns.

This is part of a webinar covering best practices on designing and building your PDI transformations and jobs for maximum speed, reuse, portability, and debugging.

Audience: Solution architects and designers, or anyone with a background in real time data ingestion, or messaging systems like JMS.

PDI offers the Fixed File Input step for reading fixed-width text files. On the output side, there is no step dedicated to this specific purpose, but fixed-width text can still be written using the existing Text file output step. This document walks you through the changes you will need to make to the default column metadata to successfully accomplish this task.

Audience: Data analysts and ETL developers who need to write fixed-width data.

Here, you will find best practices for using template-driven designs and for navigating and operating the different levels of metadata injection. We have provided an example of how to build the data-driven rule extract/transform/load (ETL) transformation and make it flexible so that it can be added to, changed, or removed without adding development cycles.

Audience: Customers in need of tips for using template-driven designs, and for building the template-driven ETL transformation.

Restartability architecture in PDI jobs and transformations plays a key role in restarting a failed ETL process where it left off. This is true whether you need to avoid duplicate entries in the target database, or you are simply seeking overall ETL efficiency and do not want to rerun processes that completed successfully in the previous run.

Audience: Pentaho ETL developers and architects, or anyone who is interested in learning PDI development patterns.

Performance and management problems are addressed here by detailing an automated method to improve performance with segmenting a single repository into several smaller repositories using PDI.

Audience: Customers looking to create a single PDI repository for maintaining many environments.

The main objective of this document is to provide information about the different options for best practices for PDI logging. Some of the things discussed here include reasons for using PDI logging, levels of logging, transformation and job logging, and debugging transformations and jobs.

Audience: Customers or developers who may be interested in using PDI logging.

This guide provides an overview of factors that can affect the performance of PDI jobs and transformations, and it provides a methodical approach to identifying and addressing bottlenecks.

Audience: Customers who seek assistance with finding and fixing bottlenecks in their PDI performance.

This document covers some best practices on integrating R with PDI, including how to install and use R with PDI and why you would want to use this setup.

Audience: Data analysts, data scientists, and PDI users who need to use the variety of statistical and machine learning tools available in the R environment.

This is a collection of best practices on using Pentaho Analyzer with Impala data sources, including how to prepare and partition data and set configurations. You will also learn about schema recommendations and settings for Analyzer.

Audience: Pentaho developers, system administrators, and architects.

This document covers some best practices on integrating Pentaho software with Cloudera QuickStart VM, including how to configure the QuickStart VM so that Pentaho can connect to it.

Audience: Pentaho developers and system architects looking to experiment with PDI and Hadoop.

PDI can be used to move objects to and from Hitachi Content Platform (HCP). Starting in PDI 8.3, we have introduced native steps that give you the ability to query HCP objects as well as read and write object metadata, allowing you to create ETL solutions for batch file processing directly in the HCP environment.

Audience: ETL developers, BI developers, storage administrators, or anyone with a background in database development, BI, or IT storage who is interested in working with files stored in HCP using PDI.