Improved Gitlab All in case of many repositories #372
Conversation
WalkthroughThe changes introduce batching to various operations that previously processed large arrays all at once. GitLab repository fetching, scheduling of repository indexing, and database updates for repository indexing status are now performed in sequential or chunked batches. These modifications affect internal control flow but do not alter any exported function or class signatures. Changes
Sequence Diagram(s)sequenceDiagram
participant Caller
participant GitLabManager
participant GitLabAPI
Caller->>GitLabManager: getGitLabReposFromConfig(config)
GitLabManager->>GitLabAPI: fetch all groups (batched, size=10)
GitLabManager->>GitLabAPI: fetch all users (batched, size=10)
loop For each group batch
GitLabManager->>GitLabAPI: fetch group projects (batch)
end
loop For each user batch
GitLabManager->>GitLabAPI: fetch user projects (batch)
end
GitLabManager-->>Caller: return repositories
sequenceDiagram
participant RepoManager
participant Scheduler
RepoManager->>RepoManager: fetchAndScheduleRepoIndexing(repos)
loop For each batch of 100 repos
RepoManager->>Scheduler: scheduleRepoIndexingBulk(batch)
end
sequenceDiagram
participant Action
participant Prisma
Action->>Action: flagReposForIndex(repoIds)
loop For each batch of 1000 IDs
Action->>Prisma: updateMany(batch, status=NEW)
end
sequenceDiagram
participant Initializer
participant Prisma
Initializer->>Initializer: syncConnections()
loop For each batch of 100 failed repo IDs
Initializer->>Prisma: updateMany(batch, status=NEW)
end
Poem
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
packages/web/src/actions.ts (1)
923-933: Good implementation of batching to prevent timeouts.The sequential batching approach effectively addresses the API timeout issues mentioned in the PR objectives. The chunk size of 1000 is reasonable for database operations.
Consider adding error handling to ensure partial failures don't leave the system in an inconsistent state:
for (let i = 0; i < repoIds.length; i += 1000) { + try { await prisma.repo.updateMany({ where: { id: { in: repoIds.slice(i, i + 1000) }, orgId: org.id, }, data: { repoIndexingStatus: RepoIndexingStatus.NEW, } }); + } catch (error) { + logger.error(`Failed to update batch ${i / 1000 + 1}: ${error}`); + throw error; // Re-throw to maintain existing error handling behavior + } }packages/web/src/initialize.ts (1)
70-81: Effective batching implementation for retry operations.The batching approach prevents timeouts when retrying failed repositories. The smaller batch size of 100 (compared to 1000 in actions.ts) is appropriate for retry operations which may be more resource-intensive.
Consider adding error handling similar to the suggestion for actions.ts to ensure robustness during batch processing.
packages/backend/src/gitlab.ts (3)
10-10: Remove unused import.The
logimport from "console" is not used anywhere in the code.-import { log } from "console";
81-81: Consider making batch size configurable.The hardcoded batch size of 10 may not be optimal for all GitLab instances. Consider making this configurable or using different batch sizes based on the instance capabilities.
- const batchSize = 10; + const batchSize = config.batchSize || 10;Also applies to: 130-130
85-120: Sequential batching may impact performance.While sequential processing prevents API overloading, it could significantly slow down the operation. Consider processing a few batches concurrently (e.g., 2-3 batches) to balance performance and API limits.
- // Process groups in batches of 10 - for (let i = 0; i < config.groups.length; i += batchSize) { - const batch = config.groups.slice(i, i + batchSize); - logger.debug(`Processing batch ${i/batchSize + 1} of ${Math.ceil(config.groups.length/batchSize)} (${batch.length} groups)`); - - const batchResults = await Promise.allSettled(batch.map(async (group) => { + // Process groups in batches of 10 with limited concurrency + const maxConcurrentBatches = 2; + const batches = []; + for (let i = 0; i < config.groups.length; i += batchSize) { + batches.push(config.groups.slice(i, i + batchSize)); + } + + for (let i = 0; i < batches.length; i += maxConcurrentBatches) { + const concurrentBatches = batches.slice(i, i + maxConcurrentBatches); + const batchPromises = concurrentBatches.map(async (batch, index) => { + logger.debug(`Processing batch ${i + index + 1} of ${batches.length} (${batch.length} groups)`); + return Promise.allSettled(batch.map(async (group) => {Also applies to: 134-169
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
packages/backend/src/gitlab.ts(3 hunks)packages/backend/src/repoManager.ts(1 hunks)packages/web/src/actions.ts(2 hunks)packages/web/src/initialize.ts(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
packages/backend/src/gitlab.ts (1)
packages/backend/src/utils.ts (2)
measure(9-17)fetchWithRetry(92-119)
🔇 Additional comments (3)
packages/backend/src/repoManager.ts (1)
168-170: Excellent batching strategy for repository scheduling.This implementation effectively addresses the core issue described in the PR objectives by preventing overwhelming the system when scheduling large numbers of repositories for indexing. The batch size of 100 aligns well with the database operation limits and job queue constraints.
The sequential processing approach is particularly important here as it prevents:
- API timeouts from GitLab
- Database connection exhaustion
- Job queue overload
packages/backend/src/gitlab.ts (2)
49-78: Excellent implementation of the all-groups/all-users feature.The logic properly handles the GitLab cloud restriction, uses retry mechanisms, and includes comprehensive error handling and logging. The approach of fetching all groups and users first, then processing them with the existing batched logic is well-designed.
89-118: Good error handling and logging implementation.The error handling properly distinguishes between 404 (not found) and other errors, and the detailed logging will be helpful for debugging issues with large repository sets. The use of
measureutility provides valuable performance insights.Also applies to: 138-166
| }); | ||
| logger.debug(`Found ${_groups.length} groups in ${groupsDuration}ms.`); | ||
|
|
||
| config.groups = _groups.map(g => g.full_path); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Avoid mutating input parameters.
Modifying the config object directly creates unexpected side effects for callers. Consider storing the discovered groups and users in local variables instead.
- config.groups = _groups.map(g => g.full_path);
+ const discoveredGroups = _groups.map(g => g.full_path);
+ config = { ...config, groups: discoveredGroups };- config.users = _users.map(u => u.username);
+ const discoveredUsers = _users.map(u => u.username);
+ config = { ...config, users: discoveredUsers };Also applies to: 69-69
🤖 Prompt for AI Agents
In packages/backend/src/gitlab.ts at lines 57 and 69, avoid directly mutating
the input parameter `config` by assigning to its properties. Instead, create
local variables to hold the groups and users data derived from `_groups` and
other sources, and use these local variables within the function. This prevents
side effects on the caller's `config` object and maintains functional purity.
| allRepos = allRepos.concat(validRepos); | ||
| notFound.orgs = notFoundOrgs; | ||
| logger.debug(`Found ${validRepos.length} valid repositories in groups.`); | ||
| logger.debug(`Not found groups: ${notFoundOrgs.join(', ')}`); | ||
| logger.debug(`These repositories will be downloaded: ${allRepos.map(repo => repo.path_with_namespace).join('\n')}`); |
There was a problem hiding this comment.
nit: synced not downloaded to be consistent with language elsewhere
|
Thanks for the change! I assume you've already tested this with larger configs? Would love to know if you've noticed a difference |
When there are many repositories on Giltab, doing actions in one hit would fail to API timeout.
This is a quick workaround/hack by using batches.
Also when there a lot of repositories, just getting the list of them might be too much.
Ideally, I would like Sourcebot to add a repository as it finds it instead of handling the entire connection sync. Otherwise this process blocks other connections from syncing, and if it fails/breaks, then it has to start over.
Note: prisma also failing when there are too many connections, not sure what is the magical number is (sometimes 1000 resulted with errors too):
Summary by CodeRabbit