New features in TeamForge 17.4

TeamForge 17.4 has a lot of new features and enhancements. Here's a list of a few release-defining new features in TeamForge 17.4.

TeamForge OAuth 2.0 authorization framework

In the traditional client-server authentication model, client applications must have access to the Resource Owner's credentials in orderto request and access protected resources on the Resource Server. Sharing Resource Owner's credentials such as usernames and passwords to third party applications poses several problems, security being one of the major ones. Frameworks such as the OAuth 2.0 and OpedID Connect help in mitigating such security risks and hence TeamForge's authentication model has been completely rebuilt on top of OpenID Connect and OAuth 2.0 Authorization frameworks.

Federated Identity Management

Federated Identity Management refers to the process of storing user credentials with an IdP (such as TeamForge, SAML and so on) and having the Service Provider (SP) rely on the IdP to validate user credentials when a user tries to access its resources.

With the new TeamForge Identity Management built on OpenID Connect (OIDC) and OAuth 2.0 authorization frameworks, TeamForge can now act as an ID Provider (IdP). As an IdP, TeamForge can authorize a third-party client application to obtain limited access to its services either on behalf of a Resource Owner (user) or on behalf of the client application itself.

TeamForge supports federated identity. By default, federated login is disable in TeamForge. Site Administrators must enable federated login and select an IdP.

Both TeamForge REST and SOAP APIs use OAuth 2.0 access tokens for authentication. Clients can obtain access tokens from the token endpoint which is located at /oauth/auth/token.

For more information, see Identity Management: Set up TeamForge OAuth 2.0 Authorization Framework.

Integrations

TeamForge-JIRA integration

This release brings a new feature for JIRA site administrators to disable active JIRA-TeamForge project mappings in bulk (Administration > Add-ons > TeamForge Associations). The intended use case revolves around staging JIRA upgrades. When a JIRA server is mirrored into a staging/testing environment, any changes to issues in projects that are mapped to TeamForge will trigger events back to TeamForge, thereby polluting the production TeamForge event data store. Use this feature to disable all production JIRA-TeamForge mappings in stage environments and then test the TeamForge Associations add-on by creating a new mapping between a staging JIRA server and TeamForge server.

For more information, see Remove TeamForge project mappings.

Nexus integration

CTF-Nexus-Integration-Plugin-2.1.1 is now available. To upgrade, see Upgrade the TeamForge-Nexus integration plugin.

TeamCity integration

TeamForge EventQ now supports TeamCity versions 9.0 and 10.0.

My Workspace

You can create one or more dashboards to categorize and view data in many different views. For instance, you may decide to create three dashboards: Dashboard A, B and C. With these three dashboards, you may choose to configure Dashboard A and B to view events from projects A and B respectively and configure Dashboard C to view just the code reviews. For more information, see My Workspace.

Configure Application tool

The Configure Application tool, added in TeamForge 17.4, makes TeamForge site level settings configurable via the user interface. This comes in handy for site administrators, who otherwise would have to work with the site-options.conf file and then recreate the TeamForge runtime for site level configuration changes. For more information, see Configure your site's settings.

The Configure Application tool is not loaded with all the available site configuration settings available in TeamForge. For settings that are not available in the Configure Application tool, Site Administrators must go the site-options.conf route as always.

Tags

The tagging functionality implemented for documents in TeamForge 17.1 has been extended to support tracker artifacts as well in TeamForge 17.4. Project members with CREATE/EDIT permissions can create new tags or add existing tags to tracker artifacts from the Create Artifact and Edit Artifact pages. You can add up to 10 tags to an artifact. You can add or remove tags to tracker artifacts when you search for artifacts within a single tracker or across trackers or during tracker mass update. You can also include "Tags" as one of the columns in the List Artifacts page.

For more information, see Create a tracker artifact and Edit a tracker artifact pages.

Tracker workflow enhancements

TeamForge 17.4 introduces Advanced Transition settings, which you can use to make specific fields as 'required' and also to show or hide specific fields when a status transition happens. Advanced Transition settings are applied when you create or edit artifacts in a tracker and only when you edit artifacts in Planning, Task, and Kanban boards. For more information, see Create a tracker workflow.

SCM/Git/Gerrit

Repository Attachment Folder

You can now select a folder to upload code review and release notes file attachments. All the files attached to the repository are stored in the selected folder. For more information, see Create a source code repository.

Highlights of TeamForge-Git integration: 17.4.6-2.13.8

  • Based on vanilla Gerritt v2.13.8. See Gerrit v2.13.8 release notes for more information.
  • Git installer enhancements to support TeamForge 17.4 installer enhancements.
  • User interface performance enhancements to better support branches and Gerrit reviews that are substantially large in size.
  • GERRIT_FORCE_HISTORY_PROTECTION site-options token is no longer supported. Remove this token from the site-options.conf file while upgrading to TeamForge 17.4 or later. History Protection is enabled by default in TeamForge 17.4 and later. However, Site Administrators can disable History Protection at the site level if need be, after which Project Administrators can choose to have History Protection enabled or disabled for individual repositories.

Highlights of TeamForge-Git integration: 17.4.11-2.13.9

  • Based on 2.13.9 vanilla Gerrit version released on 2 August, 2017. See v2.13.9 for more information.
  • Fixed the issue in which the RabbitMQ connections and threads are kept permanently in memory even if the project is not active currently.

    Cause of the issue:

    This issue is caused by the default configuration of RabbitMQ client which opens the dedicated thread pool (that spawns up to 2 * number of CPU threads by default) per connection. The issue manifests with large number of threads being spawned over time up to,
    • (number of repositories + number of projects) for threads named AMQP Connection AA.BB.CC.DD.YYYY.
    • (number of repositories + number of projects) * (2 * number of CPUs) for threads named pool-XXX-thread-Y (especially when the machine is heavily loaded and multiple changes are being reviewed and pushed to multiple projects).

    Solution:

    • Shared thread pool used by all RabbitMQ connections: It can spawn up to (factor * number of CPUs) threads where factor can be modified by setting teamforge.notificationMaxRabbitMQExecutorsFactor in gerrit.config (default value is 2) file. Threads are removed from thread pool after being idle for more than teamforge.notificationMaxRabbitMqThreadIdleTime seconds in gerrit.config (by default 3*60s) file. This addresses the issue with the pool-XXX-thread-Y threads explosion.
    • Connection Expiration: Connection to get closed after being idle for more than teamforge.notificationMaxRabbitMQConnectionIdleTime seconds in gerrit.config (default is 5*60s) file. This addresses the issue with the AMQP Connection AA.BB.CC.DD:YYYY threads explosion as they are not being quickly (marked as "not used" after teamforge.notificationMaxRabbitMQConnectionIdleTime idle time and guaranteed to be collected within (2 * teamforge.notificationMaxRabbitMQConnectionIdleTime idle time)) collected.
  • You can now search for the changes using Change-Id triplet (Repository~Ref~Change-Id). When the first 5 characters of the Repository name (Example: IdeaField) matches the Change-Id pattern, the changes searched for will be shown. This fix is sent back to open source Gerrit.

Highlights of TeamForge-Git integration: 17.4.14-2.13.9

  • For the vanilla based Gerrit release notes and the bug fixes, see Gerrit v2.13.9-11-gf25da13683 released on November 3, 2017.

  • HTTP webhooks functionality has been implemented to set up communication between TeamForge Git integration and the following products:
    • Lifecycle
    • Continuum
    • DLM 2.0
    • Any other tool that accepts HTTP notifications
    TeamForge UI for webhooks (implemented in TeamForge 17.11) is not backported. However, you can use the following products to configure webhooks:
    • CollabNet Desktop (latest development version)
    • GitEye (latest development version)
    • TeamForge WebHooks Configurator
  • Bug fixes

    Critical upstream (stable-2.13) Gerrit fix is now available for SSH clone/fetch failure when server takes more than 30 seconds to compute its operation. To handle this, a new parameter sshd.waitTimeout has been included in the gerrit.config file. By default, the value of this parameter is 30 seconds.

EventQ

EventQ Installation/Upgrade Changes

Starting from TeamForge17.4, you do not have to install EventQ separately on top of TeamForge. As long as you have your site's master configuration file (site-options.conf) specifying where your EventQ services (eventq, mongodb, redis and rabbitmq) are hosted, the TeamForge installer, by default, installs or upgrades EventQ services as part of TeamForge installation/provision.

In case you install or upgrade EventQ on a separate server, you must configure your TeamForge installation repository on the EventQ server and install EventQ using the teamforge-eventq meta RPM (yum install teamforge-eventq).

Obsolete site-options.conf tokens

As a result of EventQ installer changes, the following EventQ tokens are no longer required/supported (see Site options change log).
  • ORC_HOSTNAME
  • ORC_SSL_CA_CERT_FILE
  • ORC_PORT
  • ORC_PROXIED_PATH
  • ORC_PROTOCOL
  • ORCHESTRATE_ENABLED
  • POSTGRES_INTERFACE_IP
  • POSTGRES_INTERFACE

Remove these tokens from the site-options.conf file while upgrading to TeamForge17.4 or later.

For more information, see TeamForge17.4 install/upgrade instructions.

Datamart performance enhancement for flex fields

A new XML to non-XML conversion based solution has been implemented to redefine the flex field data storage and retrieval mechanism. This new approach improves the performance of reporting queries used for retrieving flex field data from the Datamart. For more information, see Why do I get performance issues when retrieving flex field information from Datamart?

Install / Upgrade

Changes to supported software versions
  • Postgres 9.6.1
  • James 2.3.2.1

Default values for site-options.conf tokens

Default values have been assigned to the following site-options.conf tokens and are therefore removed from the default TeamForge 17.4 site-otpions-default.conf file.

Warning: If you are upgrading from TeamForge 17.1 (or earlier) to TeamForge 17.4 (or later) and if you have been using your own values for the following tokens, you must make sure you use the same values in your site-options.conf file post upgrade to TeamForge 17.4.
# CTF Core
SCM_ADMIN_PASSWORD=$auto$
IAF_DBPASS=$auto$
SCM_DEFAULT_SHARED_SECRET=$auto$
SOAP_ANONYMOUS_SHARED_SECRET=$auto$

# Database
DATABASE_USERNAME=teamforge
DATABASE_NAME=teamforge
DATABASE_PASSWORD=$auto$
DATABASE_READ_ONLY_USER=teamforge_reader
DATABASE_READ_ONLY_PASSWORD=$auto$

# Datamart
REPORTS_DATABASE_USERNAME=teamforge_datamart
REPORTS_DATABASE_NAME=teamforge_datamart
REPORTS_DATABASE_PASSWORD=$auto$
REPORTS_DATABASE_READ_ONLY_USER=teamforge_datamart_reader
REPORTS_DATABASE_READ_ONLY_PASSWORD=$auto$

# ETL
ETL_SOAP_SHARED_SECRET=$auto$

# Gerrit
GERRIT_DATABASE_PASSWORD=$auto$

# RabbitMQ
RABBITMQ_APP_ADMIN_USER=guest
RABBITMQ_APP_ADMIN_PASSWORD=$auto$
RABBITMQ_APP_CTF_USER=ctf
RABBITMQ_APP_CTF_PASSWORD=$auto$
RABBITMQ_APP_SERVICES_USER=eventq
RABBITMQ_APP_SERVICES_PASSWORD=$auto$
RABBITMQ_PERMISSION_PUBLISHER=permission_publisher
RABBITMQ_PERMISSION_PUBLISHER_PASSWORD=$auto$

# Mongo
MONGODB_APP_DATABASE_NAME=eventq
MONGODB_ADMIN_DATABASE_NAME=admin
MONGODB_APP_ADMIN_USER=admin
MONGODB_APP_ADMIN_PASSWORD=$auto$
MONGODB_APP_BACKUP_USER=backup
MONGODB_APP_BACKUP_PASSWORD=$auto$
MONGODB_APP_SERVICES_USER=eventq
MONGODB_APP_SERVICES_PASSWORD=$auto$

# HAProxy
HAPROXY_STATS_PASSWORD=$auto$

# Binary
IAF_DBNAME=iafdb
IAF_DBUSER=iafdbusr

# James Admin (for management interface)
JAMES_ADMIN_USER=admin
JAMES_ADMIN_PASSWORD=$auto$

LISTEN_IP

By default, services bind to the IP address corresponding to the <host>:PUBLIC_FQDN token. However, you can override this using the <host>:<service>:LISTEN_IP site-options.conf token. You can use this <host>:<service>:LISTEN_IP token to control which IPs the services bind to so that you can make sure that services are not overexposed than necessary.

For example:
mypostgresserver:LISTEN_IP = 1.2.3.4
localhost:mail:LISTEN_IP = 1.2.3.4
More changes
  • The is_service_running.py script is no longer available.
  • TeamForge 17.4 installer expects the system locale to be LANG=en_US.UTF-8. TeamForge create runtime (/opt/collabnet/teamforge/bin/teamforge provision) fails otherwise.
  • The Review Board application, which was used to be installed only on the TeamForge Application Server, can now be installed on a separate server.
  • The default Review Board data directory has been changed from /opt/collabnet/reaviewboard/data to /opt/collabnet/teamforge/var/reviewboard/data in TeamForge 17.4. Make sure you restore your Review Board backup to the new data directory, which is /opt/collabnet/teamforge/var/reviewboard/data.
  • The publish/remotePublishLog is now being generated in /opt/collabnet/teamforge/log-archive/others/ as expected.
  • As PostgreSQL upgrade is taken care of by the teamforge script during teamforge provision, pgturant utility is no longer required and therefore not supported any longer.
  • CEE compatibility mode is no longer supported.
  • Warning messages displayed during the installation or upgrade are now changed from "Yellow" to "Amber" color.
  • For a list of obsolete and new site option tokens in TeamForge 17.4, see Site options change log.

snapshot.py

The snapshot.py script has been deprecated. The teamforge script subsumes the snapshot.py script's functions. For more information, see TeamForge script.